From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 873C9C6379F for ; Tue, 17 Jan 2023 04:31:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Reply-To:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hu0y+cm9QXc8eLeKxAr6GDUuCtGQzfHzWa3UbSSM814=; b=D8oJIiWzl+XUsl 6Zw461a0bgNMoiL2Uz56tugu1HrkO2fLLn6ASGWFDnDFyunGZey5A8FliCtC3A9zNpibmToMTRfun wJ0FzNaenMoL5QeZcjV3kX+nA4X7NuRQEzulZ5Q1wrEFfvZV8eff8H1hLmCJJw9M9FfatuLBRTvEX IwhKdJtYHKoqxg2GCScWGBKQS2a86/pyxxFkVBlp2OSuj+pwzhS7E35L+G5XFH57K8XS+FmqoY169 F47BT6H6LR9tFDHmV0mHXIv5PYsUJCcZairwvyn826RMPwvesuHOB7avplrJRoT3DBkVrBahOpvLf ct0G+yomFcoXF9wgT9Jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pHdc4-00Csha-1f; Tue, 17 Jan 2023 04:30:20 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pHdc0-00Csgw-1j for linux-arm-kernel@lists.infradead.org; Tue, 17 Jan 2023 04:30:17 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C3A4AB81187; Tue, 17 Jan 2023 04:30:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 762ECC433EF; Tue, 17 Jan 2023 04:30:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1673929811; bh=Vb1Z891Iid263VMKIXJGRTCq5kf4ScNrOYnm4UH6WIM=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=ntBymqeIvu+01wwe1GZzXHG9ohWlJMPnQrVXbJHXIgr5SGUHrNeJHEHfhCqlNkFoj V/VlXEREvXL5dlZgxoTzpsFsCI8XITozos6tMy9O42xiT5/M4nTUJ1fIN6fTpip9fD +iaYjwswYV3FTwvgOCvxeHhMyYVhsQuVqKPNRaUbMVEdEAPQ+xx5YX1k4Tnp0IbRxM iaoKrl47y8oPgq6i85RQzCLOTv50dPqn5j1Zvt1/HWZhNPejVWQAHNzQl9D2Mt7ywX A9RepjzgDb6PKg9oLq21S/FaTSrB0VxHdTfZ/lnSCj4cBcRzWDTQ4x6n4TbW15c7eC AchqzByqap/oQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 181B75C06D0; Mon, 16 Jan 2023 20:30:11 -0800 (PST) Date: Mon, 16 Jan 2023 20:30:11 -0800 From: "Paul E. McKenney" To: Joel Fernandes Cc: Zhouyi Zhou , "moderated list:ARM/STM32 ARCHITECTURE" , Will Deacon , Marc Zyngier , Mark Rutland , Catalin Marinas , rcu Subject: Re: arm64 torture test hotplug failures (offlining causes -EBUSY) Message-ID: <20230117043011.GD2948950@paulmck-ThinkPad-P17-Gen-1> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230116_203016_437966_1533E580 X-CRM114-Status: GOOD ( 37.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: paulmck@kernel.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Jan 17, 2023 at 12:15:07AM +0000, Joel Fernandes wrote: > On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote: > > Hi Zhouyi, > > > > On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou wrote: > > > > > [..] > > > On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes wrote: > > > > > > > > Hello, > > > > I am seeing -EBUSY returned a lot during torture_onoff() when running > > > > rcutorture on arm64. This causes hotplug failure 30% of the time. I am > > > > also seeing this in 6.1-rc kernels. I believe see this only for CPU0. > > > > > > > > This causes warnings in torture tests: > > > > [ 217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > > > > [ 221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > > > > > > > > Full kernel log here: > > > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log > > > > > > > > Any ideas on why this is happening and only for CPU 0 (presumably the > > > > boot CPU)? I'd personally need these warnings to go away for my tests > > > > as this causes rcutorture's tests to not cleanly pass for me. It > > > > appears remove_cpu() -> device_offline() is what returns the error. > > > > > > > I guess this probably because CPU 0 is the tick_do_timer_cpu in > > > nohz_full mode, which prevent that cpu from > > > going offline [1]. We have discussed this topic, but there is no > > > agreement on how to solve it yet. > > > > But I am seeing the issue in TRACE02 config which is: > > CONFIG_NO_HZ_IDLE=y > > # CONFIG_NO_HZ_FULL is not set > > > > So that is not NO_HZ_FULL: > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/ > > However, I can't seem to find the full kernel logs for that. > > > > Also, other than the TRACE02 fail, I only see the issue with configs > > with CONFIG_NO_HZ_FULL=y > > > > Can you try TRACE02 specifically, and see if you can reproduce the > > same issue on your setup? Meanwhile, I'll try to trace what is > > returning the -EBUSY. > > How about something simple like the following? (untested) > > ---8<----------------------- > > diff --git a/kernel/torture.c b/kernel/torture.c > index bc8fb361efc0..cd64110694c0 100644 > --- a/kernel/torture.c > +++ b/kernel/torture.c > @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes, > // PCI probe frequently disables hotplug during boot. > (*n_offl_attempts)--; > s = " (-EBUSY forgiven during boot)"; > + } else if (tick_nohz_full_running && ret == -EBUSY) { > + (*n_offl_attempts)--; > + s = " (-EBUSY forgiven if nohz_full is running)"; But this should be forgiven for the timekeeping CPU, not everyone, correct? Yes, I know that CPU-hotplug operations can fail, but in my testing they almost never do. This means that a new failure might well be a real bug somewhere that needs attention. Thanx, Paul > } > if (verbose) > pr_alert("%s" TORTURE_FLAG _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel