From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.3 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_PASS,USER_AGENT_MUTT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1428AC43441 for ; Mon, 12 Nov 2018 16:17:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C43442241E for ; Mon, 12 Nov 2018 16:17:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Hhuvq0xe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C43442241E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=roeck-us.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-block-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730094AbeKMCK5 (ORCPT ); Mon, 12 Nov 2018 21:10:57 -0500 Received: from mail-pg1-f193.google.com ([209.85.215.193]:39622 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729832AbeKMCK5 (ORCPT ); Mon, 12 Nov 2018 21:10:57 -0500 Received: by mail-pg1-f193.google.com with SMTP id r9-v6so4277584pgv.6; Mon, 12 Nov 2018 08:17:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=JqnyUb3NOtSqe9NhmC2WwU61rLx2veoASSw1zuSW+Hw=; b=Hhuvq0xev1ttDKcl5dYRXeSVjV7P0h6/wCJoBImY7z19CQx87TAWK4UPF0CYTIAoSq /lKJaSCRggki0Q3if8/t0GHnTsPN8zdJJsMXvw7NsLli7Swd1z0egkiprAiBaA8FSPFe aNx0mMm0RtiCJ+mZfPb6F+493+5El3dNsRJC9O3FWjWhFolpo2H5Hzc2tecpMMN8e8Du 7eWpYoKkCgafIuGcIgu+k01zoc41ipf8pIXZwjh4TGAmOEl+RAYrcGx7WbP3jXy54s0j yze/eiX0o9pq0Tu284DZffCuhE3cEamL9tQp1PwJzxTqR+NWSpJqeqz4U73DFKYHoCUf zkrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=JqnyUb3NOtSqe9NhmC2WwU61rLx2veoASSw1zuSW+Hw=; b=jXJkhHxEqe+ZGIhifVWcLy6JUONtpZiUloV7Zio5KlyT10QH8FoopYrsLJxORj/qi7 LZ2dRBQbhzYKFf0aWYjzRkd0qoWeUAYOEAS2zmc6CgFhr5jPSTNBHH8F31pf5ID+gEob /3JuR8ux3VYrkG8YN3KLMWWi3Uwz57tLzep2DTH+XwPvxcEgCk2TkoYdJRv4UZ078f0S TUD6SQUkrmCif55nMgFjr37h1FNewNkXeb7ErpD2/zuyL4kqq5B/pwjg1LpQ5rMHo4L/ wgeoNGoqiNA0BhXmo9066DU03xQcEw1RSIODa7rpB30MVYznmfAyFaQtdSzt5pmY5M+G dvhw== X-Gm-Message-State: AGRZ1gJQt9jGXIwA8enONkm7cskgM66AFt/Sih4eJep6CBgFRvXGBmae C7Dsf8q+wBoioPVJC1uy154= X-Google-Smtp-Source: AJdET5f6X8yW9fFC1K3SzFFgilXwiUmOA079LW2T1Pd/e5mG63E0GyqSRq3lnBDd9E7mS7+8yFLqYQ== X-Received: by 2002:a63:104d:: with SMTP id 13mr1345131pgq.303.1542039421462; Mon, 12 Nov 2018 08:17:01 -0800 (PST) Received: from localhost ([2600:1700:e321:62f0:329c:23ff:fee3:9d7c]) by smtp.gmail.com with ESMTPSA id b14-v6sm20560166pgn.49.2018.11.12.08.16.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 08:17:00 -0800 (PST) Date: Mon, 12 Nov 2018 08:16:58 -0800 From: Guenter Roeck To: Ming Lei Cc: Ming Lei , Jens Axboe , Peter Zijlstra , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Hannes Reinecke , Paolo Bonzini , Christoph Hellwig , "Martin K. Petersen" , "James E.J. Bottomley" , linux-scsi@vger.kernel.org Subject: Re: kobject lifetime issues in blk-mq Message-ID: <20181112161658.GA13176@roeck-us.net> References: <20181109203518.GA7130@roeck-us.net> <20181112092051.GA391@ming.t460p> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20181112092051.GA391@ming.t460p> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On Mon, Nov 12, 2018 at 05:20:52PM +0800, Ming Lei wrote: > On Fri, Nov 09, 2018 at 12:35:18PM -0800, Guenter Roeck wrote: > > Hi, > > > > Images with CONFIG_DEBUG_KOBJECT=y, CONFIG_DEBUG_KOBJECT_RELEASE=y > > enabled report kobject lifetime issues when booting an image in qemu > > using virtio-scsi-pci. > > > > Bisect points to two different commits, depending on the kernel > > configuration. > > > > 1st configuration: defconfig plus: > > > > CONFIG_VIRTIO_BLK=y > > CONFIG_SCSI_LOWLEVEL=y > > CONFIG_SCSI_VIRTIO=y > > CONFIG_VIRTIO_PCI=y > > CONFIG_VIRTIO_MMIO=y > > CONFIG_DEBUG_OBJECTS=y > > CONFIG_DEBUG_OBJECTS_FREE=y > > CONFIG_DEBUG_OBJECTS_TIMERS=y > > CONFIG_DEBUG_KOBJECT=y > > CONFIG_DEBUG_KOBJECT_RELEASE=y > > > > Bisecting this configuration points to commit b5b6e8c8d3b4 ("scsi: > > virtio_scsi: fix IO hang caused by automatic irq vector affinity") > > as the culprit. This commit enforces the use of blk-mq for virtio_scsi; > > the bisect log is therefore a bit misleading. > > > > 2nd configuration: As above, plus: > > CONFIG_SCSI_MQ_DEFAULT=y > > > > With this configuration, bisect points to commit 7ea5fe31c12d > > ("blk-mq: make lifetime consitent between q/ctx and its kobject"). > > > > The qemu command line used to reproduce the problem is as follows. > > The qemu version does not seem to matter (I tested with qemu versions > > 2.5, 2.11, and 3.0). > > > > qemu-system-x86_64 \ > > -kernel arch/x86/boot/bzImage \ > > -device virtio-scsi-pci,id=scsi \ > > -device scsi-hd,bus=scsi.0,drive=d0 \ > > -drive file=./wheezy.img,format=raw,if=none,id=d0 \ > > -snapshot \ > > -m 2G -smp 4 -enable-kvm \ > > -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic \ > > -nographic -monitor none \ > > -no-reboot \ > > -append "console=ttyS0 root=/dev/sda earlyprintk=serial panic_on_warn=1 panic=-1" > > > > Root file system: > > https://storage.googleapis.com/syzkaller/wheezy.img > > or: > > https://github.com/groeck/linux-build-test/blob/master/rootfs/x86_64/rootfs.ext2.gz > > > > It seems that any root file system can be used to reproduce the problem. > > Also, the problem is not limited to virtio-scsi-pci. I also tried > > am53c974, lsi53c810, and lsi53c895a (after enabling the respective > > drivers), with the same result. > > > > Overall, this suggests that the problem is related to blk-mq and was > > indeed introduced with commit 7ea5fe31c12d. > > > > For reference, I attached an actual crash log as well as a set of bisect > > logs below. > > > > Unfortunately, my understanding of blk-mq is not good enough to find the > > underlying problem and suggest a fix. Please let me know if there is > > anything else I can do to help fixing the problem. > > > > Note that the problem was originally reported by syzbot running a test > > on chromeos-4.14. It may well be that the problem has already been > > reported and is being worked on. If so, please apologize the noise. > > > > Thanks, > > Guenter > > > > --- > > [ 8.700038] ------------[ cut here ]------------ > > [ 8.700376] ODEBUG: free active (active state 0) object type: timer_list hint: delayed_work_timer_fn+0x0/0x10 > > [ 8.701032] WARNING: CPU: 0 PID: 0 at lib/debugobjects.c:329 debug_print_object+0x6a/0x80 > > [ 8.701032] Kernel panic - not syncing: panic_on_warn set ... > > [ 8.701032] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.20.0-rc1 #30 > > [ 8.701032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 > > [ 8.701032] Call Trace: > > [ 8.701032] > > [ 8.701032] dump_stack+0x46/0x5b > > [ 8.701032] panic+0xf3/0x246 > > [ 8.701032] ? debug_print_object+0x6a/0x80 > > [ 8.701032] __warn+0xeb/0xf0 > > [ 8.701032] ? debug_print_object+0x6a/0x80 > > [ 8.701032] ? debug_print_object+0x6a/0x80 > > [ 8.701032] report_bug+0xb1/0x120 > > [ 8.701032] fixup_bug.part.10+0x13/0x30 > > [ 8.701032] do_error_trap+0x8f/0xb0 > > [ 8.701032] do_invalid_op+0x31/0x40 > > [ 8.701032] ? debug_print_object+0x6a/0x80 > > [ 8.701032] invalid_op+0x14/0x20 > > [ 8.701032] RIP: 0010:debug_print_object+0x6a/0x80 > > [ 8.701032] Code: 8b 43 10 83 c2 01 8b 4b 14 89 15 c1 e2 78 01 4c 8b 45 00 4c > > 89 e6 48 c7 c7 f0 ca 41 a6 48 8b 14 c5 a0 f7 24 a6 e8 06 8e cc ff <0f> 0b 5b 83 > > 05 70 47 1a 01 01 5d 41 5c c3 83 05 65 47 1a 01 01 c3 > > [ 8.701032] RSP: 0018:ffff89b9fda03e48 EFLAGS: 00010082 > > [ 8.701032] RAX: 0000000000000000 RBX: ffff89b9fd49d0c8 RCX: ffffffffa6646e18 > > [ 8.701032] RDX: 0000000000000001 RSI: 0000000000000082 RDI: ffffffffa6cc596c > > [ 8.701032] RBP: ffffffffa664a5c0 R08: 79616c6564203a74 R09: 5f6b726f775f6465 > > [ 8.701032] R10: ffff89b9fda03f10 R11: 3178302f3078302b R12: ffffffffa6403e22 > > [ 8.701032] R13: 0000000000000001 R14: ffffffffa6d4d458 R15: ffff89b9fc4c8780 > > [ 8.701032] ? debug_print_object+0x6a/0x80 > > [ 8.701032] debug_check_no_obj_freed+0x1b8/0x1ea > > [ 8.701032] ? rcu_process_callbacks+0x2a7/0x750 > > [ 8.701032] kmem_cache_free+0x6e/0x1a0 > > [ 8.701032] ? __blk_release_queue+0x150/0x150 > > [ 8.701032] rcu_process_callbacks+0x2a7/0x750 > > [ 8.701032] __do_softirq+0xf2/0x2c7 > > [ 8.701032] irq_exit+0xb7/0xc0 > > [ 8.701032] smp_apic_timer_interrupt+0x67/0x130 > > [ 8.701032] apic_timer_interrupt+0xf/0x20 > > [ 8.701032] > > [ 8.701032] RIP: 0010:default_idle+0x12/0x140 > > [ 8.701032] Code: fe ff ff c6 43 08 00 fb eb b0 31 c0 eb b4 e8 65 c8 60 ff 90 > > 90 90 90 90 41 54 55 53 65 8b 2d 15 c4 3b 5a 66 66 66 66 90 fb f4 <65> 8b 2d 07 > > c4 3b 5a 66 66 66 66 90 5b 5d 41 5c c3 65 8b 05 f6 c3 > > [ 8.701032] RSP: 0018:ffffffffa6603e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > > [ 8.701032] RAX: ffffffffa5c52d10 RBX: 0000000000000000 RCX: 0000000000000000 > > [ 8.701032] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000000020605092a > > [ 8.701032] RBP: 0000000000000000 R08: ffffffffcff7766a R09: ffff89b9fda20b00 > > [ 8.701032] R10: 0000000000000000 R11: 0000000000002400 R12: ffffffffa6611740 > > [ 8.701032] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffa6611740 > > [ 8.701032] ? __sched_text_end+0x5/0x5 > > [ 8.701032] do_idle+0x19c/0x230 > > [ 8.701032] cpu_startup_entry+0x14/0x20 > > [ 8.701032] start_kernel+0x491/0x4b1 > > [ 8.701032] secondary_startup_64+0xa4/0xb0 > > [ 8.701032] Kernel Offset: 0x24200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > That is simply caused by enabling DEBUG_KOBJECT_RELEASE itself, which > tries to do kboject_release() after delaying a while. > > Wrt. this issue, q->mq_kobj is embedded in 'request_queue' whose > instance is freed in release handler of q->kobj. So when one instance > of 'request_queue' is freed, DEBUG_KOBJECT_RELEASE still expects that > the q->mq_kobj is active. > > The release handler of q->mq_kobj is NOP actually, so in reality it won't > be one issue at all. > So what is the solution ? Either this is an incorrect use of kobjects, or DEBUG_KOBJECT_RELEASE is broken and needs to be marked as such (or fixed, but I would have no idea how to do that). I'll be happy to send a patch marking DEBUG_KOBJECT_RELEASE as broken, but I suspect that it may be rejected with prejudice. Thanks, Guenter