From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90177C07E95 for ; Fri, 16 Jul 2021 16:23:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5E77F613CC for ; Fri, 16 Jul 2021 16:23:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229534AbhGPQ0i (ORCPT ); Fri, 16 Jul 2021 12:26:38 -0400 Received: from mail.kernel.org ([198.145.29.99]:41690 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229462AbhGPQ0g (ORCPT ); Fri, 16 Jul 2021 12:26:36 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 7C039613BB; Fri, 16 Jul 2021 16:23:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1626452621; bh=POGqhedhygshA0XZUU3Op4Fw0hWujqwUgWAR4cjt94Y=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=EwkBhWLmwcpbxMaqHrlKgtHbEqpAtFpEbjYJlxCouiiZuKnoWCXpO4lxG3VGy6h9s ERGaQGaLHy4Ob9ssKIg0+E3ekeap/Ux2QceKuL81cPqRSEWFVkoYVgB97laQbkMICh 0XtKW1fcUaUe7KaRzzIelL3vMKMyIshtF+I85edLy02QFM6dAMSq4Ws02IYbCtN7Bz HRZMzmBTIr8OJQR/SXSpju7q0kuGnLu/AtRltkc8obkI7CMG3PUJwrLc+/rJZ74hxz phlgQyi2uVGULEKgLddAUrLYUGnTWM8y8Ai713UtnkMJNuzfE0j4mMiO6viWvtZwoQ yoafQnxtR82jw== Date: Fri, 16 Jul 2021 09:23:40 -0700 From: "Darrick J. Wong" To: Roman Gushchin Cc: Boyang Xue , Matthew Wilcox , Jan Kara , linux-fsdevel@vger.kernel.org Subject: Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash Message-ID: <20210716162340.GY22357@magnolia> References: <20210714092639.GB9457@quack2.suse.cz> <20210715171050.GB22357@magnolia> <20210715222812.GW22357@magnolia> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210715222812.GW22357@magnolia> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Thu, Jul 15, 2021 at 03:28:12PM -0700, Darrick J. Wong wrote: > On Thu, Jul 15, 2021 at 01:08:15PM -0700, Roman Gushchin wrote: > > On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote: > > > On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote: > > > > On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox wrote: > > > > > > > > > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote: > > > > > > It's unclear to me that where to find the required address in the > > > > > > addr2line command line, i.e. > > > > > > > > > > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux > > > > > > > > > > > > > > > > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394 > > > > > > > > > > > > > Thanks! The result is the same as the > > > > > > > > addr2line -i -e > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux > > > > FFFF8000102D6DD0 > > > > > > > > But this script is very handy. > > > > > > > > # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux > > > > cleanup_offlin > > > > e_cgwbs_workfn+0x320/0x394 > > > > cleanup_offline_cgwbs_workfn+0x320/0x394: > > > > arch_atomic64_fetch_add_unless at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265 > > > > (inlined by) arch_atomic64_add_unless at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290 > > > > (inlined by) atomic64_add_unless at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149 > > > > (inlined by) atomic_long_add_unless at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491 > > > > (inlined by) percpu_ref_tryget_many at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247 > > > > (inlined by) percpu_ref_tryget at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266 > > > > (inlined by) wb_tryget at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227 > > > > (inlined by) wb_tryget at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224 > > > > (inlined by) cleanup_offline_cgwbs_workfn at > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679 > > > > > > > > # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c > > > > ``` > > > > static void cleanup_offline_cgwbs_workfn(struct work_struct *work) > > > > { > > > > struct bdi_writeback *wb; > > > > LIST_HEAD(processed); > > > > > > > > spin_lock_irq(&cgwb_lock); > > > > > > > > while (!list_empty(&offline_cgwbs)) { > > > > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback, > > > > offline_node); > > > > list_move(&wb->offline_node, &processed); > > > > > > > > /* > > > > * If wb is dirty, cleaning up the writeback by switching > > > > * attached inodes will result in an effective removal of any > > > > * bandwidth restrictions, which isn't the goal. Instead, > > > > * it can be postponed until the next time, when all io > > > > * will be likely completed. If in the meantime some inodes > > > > * will get re-dirtied, they should be eventually switched to > > > > * a new cgwb. > > > > */ > > > > if (wb_has_dirty_io(wb)) > > > > continue; > > > > > > > > if (!wb_tryget(wb)) <=== line#679 > > > > continue; > > > > > > > > spin_unlock_irq(&cgwb_lock); > > > > while (cleanup_offline_cgwb(wb)) > > > > cond_resched(); > > > > spin_lock_irq(&cgwb_lock); > > > > > > > > wb_put(wb); > > > > } > > > > > > > > if (!list_empty(&processed)) > > > > list_splice_tail(&processed, &offline_cgwbs); > > > > > > > > spin_unlock_irq(&cgwb_lock); > > > > } > > > > ``` > > > > > > > > BTW, this bug can be only reproduced on a non-debug production built > > > > kernel (a.k.a kernel rpm package), it's not reproducible on a debug > > > > build with various debug configuration enabled (a.k.a kernel-debug rpm > > > > package) > > > > > > FWIW I've also seen this regularly on x86_64 kernels on ext4 with all > > > default mkfs settings when running generic/256. > > > > Oh, that's a useful information, thank you! > > > > Btw, would you mind to give a patch from an earlier message in the thread > > a test? I'd highly appreciate it. > > > > Thanks! > > Will do. fstests passed here, so Tested-by: Darrick J. Wong --D > > --D