From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47710C433EF for ; Wed, 23 Feb 2022 00:54:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Reply-To:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SrVE7sElZ5c3F9878aIBsyfSglwCCOtBVA1dz4E7emM=; b=IcXtxJ0eixbbi6 VqpSp28AUXVDtrbKwAFv/N7rsu72eOsF5MNZnhKaEqbGkn6oss8/kVI5PRzwIoQy66ZZIbKwliUs2 VKfjK9YARHEBwKYGM8yzvGjEqwLNX28+GpdkVN1cG3qUSepxHoWassOXhm65BHw1hPJmFdDufMzeT k468EmWmg2yMSpd+QKPKeL1+Sy0t+UbuVEvsA3du7GdeRqxgEDQr8F+S5GdRvWVNdaEznd1gNYBei GmQEfi1JE/8EO58bARZ2HHknDYY5CwgQECz4NzTkfHMlMidaasCqgTGV6oJaTQ0wWMiL7il51EVFG UBHQ3y4uQ+GjO/SQARsQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMftO-00C2Bq-Li; Wed, 23 Feb 2022 00:52:30 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMftK-00C2BA-9z; Wed, 23 Feb 2022 00:52:29 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 00FD161353; Wed, 23 Feb 2022 00:52:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5EFB7C340E8; Wed, 23 Feb 2022 00:52:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1645577544; bh=wtszXTGLCE0YIbPMI7+Az+4DkUro2sZGjhHJa80rl0Q=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=q1G+S73/vN/oCIsi+w40OW6gDaZM8xjyBH+6XbFLG2HSCA9DbPCuHBeL1l/RwN4SV vqRo2ENyvQUl+uedPgsQTpvK9zuLWKLax21eJOiuidw6eau0BO7+oxNTet1wnPIJxS jMo9PUqMmUJpiP122fMR/4waS/pkhAuXnFEJcEiiz7sdhPx26FxWXB+iUhe2Rmjb2A kyOrdN4NFBB8mkqA6GHKVPBcYb119lmD/CJc+VaSRhR3hZNAtnJeiJkQ5bHCiCEjee 8XFV7+28+H3jHkeuZJd2MSi5XAe2w5tvqX2H98qYIpC6SXvlts26+jX4q7LscJXmH0 fgtPPqja5nYqw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EF80B5C0521; Tue, 22 Feb 2022 16:52:23 -0800 (PST) Date: Tue, 22 Feb 2022 16:52:23 -0800 From: "Paul E. McKenney" To: Padmanabha Srinivasaiah Cc: josh@joshtriplett.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, joel@joelfernandes.org, rcu@vger.kernel.org, linux-kernel@vger.kernel.org, tj@kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH] rcu-tasks: Fix race in schedule and flush work Message-ID: <20220223005223.GM4285@paulmck-ThinkPad-P17-Gen-1> References: <20220217152520.18972-1-treasure4paddy@gmail.com> <20220217174330.GR4285@paulmck-ThinkPad-P17-Gen-1> <20220220183517.GA3164@pswork> <20220220200425.GB4285@paulmck-ThinkPad-P17-Gen-1> <20220222220629.GA2334@pswork> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220222220629.GA2334@pswork> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_165226_453096_322E7EBC X-CRM114-Status: GOOD ( 24.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: paulmck@kernel.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Feb 22, 2022 at 11:06:29PM +0100, Padmanabha Srinivasaiah wrote: > On Sun, Feb 20, 2022 at 12:04:25PM -0800, Paul E. McKenney wrote: > > On Sun, Feb 20, 2022 at 07:35:17PM +0100, Padmanabha Srinivasaiah wrote: > > > On Thu, Feb 17, 2022 at 09:43:30AM -0800, Paul E. McKenney wrote: > > > > On Thu, Feb 17, 2022 at 04:25:19PM +0100, Padmanabha Srinivasaiah wrote: > > > > > While booting secondary CPUs, cpus_read_[lock/unlock] not keeping online > > > > > cpumask stable. The transient online mask results in below calltrace. > > > > > > > > > > [ 0.324121] CPU1: Booted secondary processor 0x0000000001 [0x410fd083] > > > > > [ 0.346652] Detected PIPT I-cache on CPU2 > > > > > [ 0.347212] CPU2: Booted secondary processor 0x0000000002 [0x410fd083] > > > > > [ 0.377255] Detected PIPT I-cache on CPU3 > > > > > [ 0.377823] CPU3: Booted secondary processor 0x0000000003 [0x410fd083] > > > > > [ 0.379040] ------------[ cut here ]------------ > > > > > [ 0.383662] WARNING: CPU: 0 PID: 10 at kernel/workqueue.c:3084 __flush_work+0x12c/0x138 > > > > > [ 0.384850] Modules linked in: > > > > > [ 0.385403] CPU: 0 PID: 10 Comm: rcu_tasks_rude_ Not tainted 5.17.0-rc3-v8+ #13 > > > > > [ 0.386473] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT) > > > > > [ 0.387289] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) > > > > > [ 0.388308] pc : __flush_work+0x12c/0x138 > > > > > [ 0.388970] lr : __flush_work+0x80/0x138 > > > > > [ 0.389620] sp : ffffffc00aaf3c60 > > > > > [ 0.390139] x29: ffffffc00aaf3d20 x28: ffffffc009c16af0 x27: ffffff80f761df48 > > > > > [ 0.391316] x26: 0000000000000004 x25: 0000000000000003 x24: 0000000000000100 > > > > > [ 0.392493] x23: ffffffffffffffff x22: ffffffc009c16b10 x21: ffffffc009c16b28 > > > > > [ 0.393668] x20: ffffffc009e53861 x19: ffffff80f77fbf40 x18: 00000000d744fcc9 > > > > > [ 0.394842] x17: 000000000000000b x16: 00000000000001c2 x15: ffffffc009e57550 > > > > > [ 0.396016] x14: 0000000000000000 x13: ffffffffffffffff x12: 0000000100000000 > > > > > [ 0.397190] x11: 0000000000000462 x10: ffffff8040258008 x9 : 0000000100000000 > > > > > [ 0.398364] x8 : 0000000000000000 x7 : ffffffc0093c8bf4 x6 : 0000000000000000 > > > > > [ 0.399538] x5 : 0000000000000000 x4 : ffffffc00a976e40 x3 : ffffffc00810444c > > > > > [ 0.400711] x2 : 0000000000000004 x1 : 0000000000000000 x0 : 0000000000000000 > > > > > [ 0.401886] Call trace: > > > > > [ 0.402309] __flush_work+0x12c/0x138 > > > > > [ 0.402941] schedule_on_each_cpu+0x228/0x278 > > > > > [ 0.403693] rcu_tasks_rude_wait_gp+0x130/0x144 > > > > > [ 0.404502] rcu_tasks_kthread+0x220/0x254 > > > > > [ 0.405264] kthread+0x174/0x1ac > > > > > [ 0.405837] ret_from_fork+0x10/0x20 > > > > > [ 0.406456] irq event stamp: 102 > > > > > [ 0.406966] hardirqs last enabled at (101): [] _raw_spin_unlock_irq+0x78/0xb4 > > > > > [ 0.408304] hardirqs last disabled at (102): [] el1_dbg+0x24/0x5c > > > > > [ 0.409410] softirqs last enabled at (54): [] local_bh_enable+0xc/0x2c > > > > > [ 0.410645] softirqs last disabled at (50): [] local_bh_disable+0xc/0x2c > > > > > [ 0.411890] ---[ end trace 0000000000000000 ]--- > > > > > [ 0.413000] smp: Brought up 1 node, 4 CPUs > > > > > [ 0.413762] SMP: Total of 4 processors activated. > > > > > [ 0.414566] CPU features: detected: 32-bit EL0 Support > > > > > [ 0.415414] CPU features: detected: 32-bit EL1 Support > > > > > [ 0.416278] CPU features: detected: CRC32 instructions > > > > > [ 0.447021] Callback from call_rcu_tasks_rude() invoked. > > > > > [ 0.506693] Callback from call_rcu_tasks() invoked. > > > > > > > > > > Link: https://lore.kernel.org/lkml/20220210184319.25009-1-treasure4paddy@gmail.com/T/ > > > > > Suggested-by: Paul E. McKenney > > > > > Signed-off-by: Padmanabha Srinivasaiah > > > > > > > > Thank you, Padmanabha! I have queued this with additional explanation > > > > in the commit log as shown below. > > > > > > Thanks paul. > > > > > > > > If you have not already tested this by repeatedly rebooting, for > > > > example continuously in an overnight test, could you please do so? > > > > (I am assuming that your system can do an automated reboot loop, so no, > > > > I am not asking you to stay up all night!) > > > > > > > Yes, tested and issue doesn't appear. > > > > Very good, thank you! > > > > May I add your Tested-by? > > > > Thanx, Paul > > > Sure paul. Once again thanks for the suggestion. > > Tested-by: Padmanabha Srinivasaiah Thank you! I will apply this on the next rebase. Thanx, Paul _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel