From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0F5DBC46467 for ; Mon, 16 Jan 2023 18:04:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=A8C6flHkSLqDhYMAyS3HxTnQeLMT/I9KtqptvSr1NOE=; b=19XWB32EIkkqSd ar+kr0TDseEzAsdNLiJda7Qq8gLONUH+915zWy2zwZMi670oadQCahyp3+nGvQaKd9ROa2L+OPxQZ b1L7dIpPHeJjVRPmFLDa3RBBhkBYzmqaWYUU93QMtU2c8LQFhUxeB6v3PwAUP3eNAHsjJ0YRy0HSs XQXLB4hX+khIxAysoVL8gw4/oxJa6vNoTvb8PGiDzuYP01DEfNwd1PNAi2eCFuPmtpQnxCnflzKK1 tsWmn7IwKxpf3Adba1U4zVIqdWI2itSxmh6aEOQFxS9duMIZYhoWlMrkHOFDVBAOZ0JkHoU/3dv33 anCoXwa86iOOExIn3FgA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pHTpM-00BdRD-Jb; Mon, 16 Jan 2023 18:03:24 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pHTpI-00BdQ0-PE for linux-arm-kernel@lists.infradead.org; Mon, 16 Jan 2023 18:03:22 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 750D1B810A2; Mon, 16 Jan 2023 18:03:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1FFF6C433EF; Mon, 16 Jan 2023 18:03:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1673892197; bh=NFPsiNZYg4maqAu/06zPYzHMj1FCUnnXkDYxZ2MpxEc=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=K2daj6F/sWXHBc1T1hyseidPZRFaUN+hNWgOF1yR1ig6Z18wiiUFRdGTLxVQ2/c2F AC6LZzwCMgPaodpgyYdm/TH5jOup/AwTzwH7SwRypI51Mp0kOPy+q4DmKWhybLqPOM wIIob3R4jm70BkM5/MF/9S0WwV/YjBcm5OMRk/v8JZXk+r3flIjMYdP9oEUnY6hdpo uCNSDBR8fkVlRI/7HNJ1M9t+LVaNzMWEyK/4i+r8YTretW0lJqLYf1Th0pIN5xQFxN 4CNkEm2tGzv0W4thafzYbaj2evi26/dhNGc2aIb5d5SG3kG5G6IMOOEMzMp2UHF2Eg FLXCd+q9ucmbg== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pHTpC-002ELD-RM; Mon, 16 Jan 2023 18:03:14 +0000 Date: Mon, 16 Jan 2023 18:03:14 +0000 Message-ID: <864jsqnyql.wl-maz@kernel.org> From: Marc Zyngier To: Joel Fernandes Cc: "moderated list:ARM/STM32 ARCHITECTURE" , Will Deacon , Mark Rutland , Catalin Marinas , rcu , "Paul E. McKenney" Subject: Re: arm64 torture test hotplug failures (offlining causes -EBUSY) In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: joel@joelfernandes.org, linux-arm-kernel@lists.infradead.org, will@kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, rcu@vger.kernel.org, paulmck@kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230116_100321_154573_15B4C32B X-CRM114-Status: GOOD ( 19.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Joel, On Mon, 16 Jan 2023 17:03:31 +0000, Joel Fernandes wrote: > > Hello, > I am seeing -EBUSY returned a lot during torture_onoff() when running > rcutorture on arm64. This causes hotplug failure 30% of the time. I am > also seeing this in 6.1-rc kernels. I believe see this only for CPU0. > > This causes warnings in torture tests: > [ 217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > [ 221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > > Full kernel log here: > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log > > Any ideas on why this is happening and only for CPU 0 (presumably the > boot CPU)? I'd personally need these warnings to go away for my tests > as this causes rcutorture's tests to not cleanly pass for me. It > appears remove_cpu() -> device_offline() is what returns the error. I've taken your kernel for a ride as a KVM guest (probably similar to what you are doing), and saw the same thing (CPU0 not offlining): [ 64.555845] Detected VIPT I-cache on CPU4 [ 64.556146] GICv3: CPU4: found redistributor 4 region 0:0x000000003ff70000 [ 64.556689] CPU4: Booted secondary processor 0x0000000004 [0x612f0290] [ 69.823670] rcu-torture:torture_onoff task: offline 0 failed: errno -16 [ 73.991960] psci: CPU7 killed (polled 0 ms) [ 74.239626] rcu-torture: rcu_torture_read_exit: Start of episode [ 74.243863] rcu-torture: rcu_torture_read_exit: End of episode I then tried v6.2-rc4 with defconfig + RCU_TORTURE and your command line, and CPU0 does seem to hotplug off correctly: [ 47.217109] psci: CPU3 killed (polled 0 ms) [ 52.241009] Detected VIPT I-cache on CPU3 [ 52.241227] cacheinfo: Unable to detect cache hierarchy for CPU 3 [ 52.241481] GICv3: CPU3: found redistributor 3 region 0:0x000000003ff50000 [ 52.241849] CPU3: Booted secondary processor 0x0000000003 [0x612f0290] [ 56.337011] psci: CPU0 killed (polled 0 ms) [...] [ 121.090339] rcu-torture: Free-Block Circulation: 922 920 919 918 917 916 914 913 912 911 0 [ 125.574311] Detected VIPT I-cache on CPU0 [ 125.574557] cacheinfo: Unable to detect cache hierarchy for CPU 0 [ 125.574901] GICv3: CPU0: found redistributor 0 region 0:0x000000003fef0000 [ 125.575322] CPU0: Booted secondary processor 0x0000000000 [0x612f0290] [ 130.176893] rcu-torture: rcu_torture_read_exit: Start of episode [ 130.317001] psci: CPU0 killed (polled 0 ms) [...] [ 225.588999] Detected VIPT I-cache on CPU0 [ 225.589224] cacheinfo: Unable to detect cache hierarchy for CPU 0 [ 225.589535] GICv3: CPU0: found redistributor 0 region 0:0x000000003fef0000 [ 225.589946] CPU0: Booted secondary processor 0x0000000000 [0x612f0290] No such error is being reported. Is there anything special in your config that would help triggering this with the current tip of tree? Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel