From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C52D1EDE9B4 for ; Tue, 10 Sep 2024 20:17:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LDHAeuFV7WeNuDu9l4ZAcORf6fGyjQ1eaKwYh+0vMpg=; b=grfFPhQb+X56Xa8qEScarV7CeO /6rgmyetBnflJ8js0f+bgQjYxmcpAP/LmTwEtxQJN8bzyvubE/qu+USl6zA9zyvkUO1sJ3QB7taDF 4Wo9hP7DxaKSgwbZO0bzAOX0ZB3uL75joIQFzXUBeq0t/vs9w9whDcNm0TN7ADdxAwM5P+XqH3BMF ZFUXPWjBiU42l6YOsvzwWRVhK2KLo2FwQNMl0mZ804f1/qrsQ41Y1c5YuUquYIkgVsfDkqD0Hentn gDj9mSe19LhBaqIgAwyTRq/BNQMZBv1iK5ujT3rZTyIB5I4FY4HKcVh6g7OYEdKyX9zmYeq9KrLAf GdHHpykQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1so7Ip-000000070BE-1qSA; Tue, 10 Sep 2024 20:17:31 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1so7Hm-0000000703h-2s6v for linux-arm-kernel@lists.infradead.org; Tue, 10 Sep 2024 20:16:29 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id F0F635C0534; Tue, 10 Sep 2024 20:16:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6506C4CEC3; Tue, 10 Sep 2024 20:16:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725999385; bh=HJYKcbje20h/WDUJRQJAyfWugUsdTUiUzb7xkzJpOL8=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=UEoSdl4SgKUVcPgxQMv7y6264JIMhvyy35scTQwfs4CQMRmxFffZaltYEPLGWBwcB QFvyj4cxTR0+gPX7XLlaj2MaFTpKPBZsa0NiFG/LGSJyPozCJNUzu4dgqbwihB4pBh 5wjWsBOI1oR0pImu66JZlnMpkfTvZlDb53lD5IUhdUXyChbhWRBaNbGPgCmcg54TM1 8aTGI4kttydl499NhLZFRosgcPdTBNmGaYBL+VoTLLTTZp6TymEaJSeZ0zOa6U5cWf wtOUG706dDOfteGwi4MHCg14vi1vbEhS4ZCcLtwEoZXipGy3MeF08Zg3h4km0YLQwf Ci2FwDYfu7G6g== Date: Tue, 10 Sep 2024 21:16:20 +0100 From: Mark Brown To: linux-kernel@vger.kernel.org Cc: linux-tip-commits@vger.kernel.org, "Darrick J. Wong" , Thomas Gleixner , "Peter Zijlstra (Intel)" , x86@kernel.org, Mark Rutland , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Aishwarya.TCV@arm.com Subject: Re: [tip: locking/urgent] jump_label: Fix static_key_slow_dec() yet again Message-ID: References: <875xsc4ehr.ffs@tglx> <172563367463.2215.5542972042769938731.tip-bot2@tip-bot2> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="NYw/P0L4bhK7Sn5+" Content-Disposition: inline In-Reply-To: <172563367463.2215.5542972042769938731.tip-bot2@tip-bot2> X-Cookie: Not every question deserves an answer. X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240910_131626_884580_6FD8A381 X-CRM114-Status: GOOD ( 15.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org --NYw/P0L4bhK7Sn5+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 06, 2024 at 02:41:14PM -0000, tip-bot2 for Peter Zijlstra wrote: > The following commit has been merged into the locking/urgent branch of ti= p: > jump_label: Fix static_key_slow_dec() yet again >=20 > While commit 83ab38ef0a0b ("jump_label: Fix concurrency issues in > static_key_slow_dec()") fixed one problem, it created yet another, > notably the following is now possible: This patch, which is now in -next appears to have caused the KVM unit tests to start exploding badly on some arm64 systems (at least N1SDP and Cavium TX2). I've bisected the issue, but not analyzed it at all beyond noting that the commit looks relevant to the failure. None of the other tests we run on these platforms seem to trigger the issue. Before producing any output the tests trigger a warning: <4>[ 17.303495] ------------[ cut here ]------------ <4>[ 17.308364] WARNING: CPU: 1 PID: 279 at kernel/jump_label.c:266 stati= c_key_dec+0x68/0x74 <4>[ 17.316706] Modules linked in: crct10dif_ce arm_spe_pmu coresight_rep= licator coresight_funnel coresight_tmc coresight_stm stm_core coresight_tpi= u arm_cmn coresight cfg80211 rfkill fuse dm_mod ip_tables x_tables ipv6 <4>[ 17.336080] CPU: 1 UID: 0 PID: 279 Comm: qemu-system-aar Not tainted = 6.11.0-rc7-00006-g3a0c7230588b #10 <4>[ 17.345719] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTY= PE=3D--) <4>[ 17.352927] pc : static_key_dec+0x68/0x74 <4>[ 17.357183] lr : __static_key_slow_dec_cpuslocked+0x24/0x84 <4>[ 17.363003] sp : ffff800083c8ba80 =2E... <4>[ 17.440381] Call trace: <4>[ 17.443074] static_key_dec+0x68/0x74 <4>[ 17.446984] static_key_slow_dec+0x2c/0x80 <4>[ 17.451327] preempt_notifier_dec+0x18/0x24 <4>[ 17.455759] kvm_destroy_vm+0x208/0x2b0 <4>[ 17.459845] kvm_vm_release+0x80/0xb0 <4>[ 17.463754] __fput+0xcc/0x2d4 <4>[ 17.467057] ____fput+0x10/0x1c <4>[ 17.470446] task_work_run+0x78/0xd4 <4>[ 17.474268] do_exit+0x2c8/0x90c then the test times out and all the remaining cores splat on: 4>[ 18.067930] registering preempt_notifier while notifiers disabled <4>[ 18.067935] WARNING: CPU: 2 PID: 470 at kernel/sched/core.c:4729 pree= mpt_notifier_register+0x24/0x58 The bisect seems to converge fairly smoothly: git bisect start # status: waiting for both good and bad commits # bad: [6708132e80a2ced620bde9b9c36e426183544a23] Add linux-next specific f= iles for 20240910 git bisect bad 6708132e80a2ced620bde9b9c36e426183544a23 # status: waiting for good commit(s), bad commit known # good: [028ac237a57e1bcb07c7130b11527c0e025e4bef] Merge branch 'for-linux-= next-fixes' of https://gitlab.freedesktop.org/drm/misc/kernel.git git bisect good 028ac237a57e1bcb07c7130b11527c0e025e4bef # good: [b66d58fce82c825b3dbb57a46b9a74f081ef7ec7] Merge branch 'main' of g= it://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git git bisect good b66d58fce82c825b3dbb57a46b9a74f081ef7ec7 # good: [a636a90415dbc59f005369e3053996f859f0af50] Merge branch 'for-next' = of git://git.kernel.org/pub/scm/linux/kernel/git/libata/linux git bisect good a636a90415dbc59f005369e3053996f859f0af50 # bad: [8e5ac35ddecbeddce79e915c226baaf577a2be6e] Merge branch 'driver-core= -next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.= git git bisect bad 8e5ac35ddecbeddce79e915c226baaf577a2be6e # bad: [1bcadc80ec6a46fb7193999935aaa299b4916569] Merge branch 'master' of = git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git git bisect bad 1bcadc80ec6a46fb7193999935aaa299b4916569 # good: [c2d0e416bdd9c83db3c9bb1f19433d5ba34e18c2] Merge branch 'for-next' = of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git git bisect good c2d0e416bdd9c83db3c9bb1f19433d5ba34e18c2 # bad: [723da3b00882e1d13038fc48ba2602af9de4dd2e] Merge branch into tip/mas= ter: 'locking/core' git bisect bad 723da3b00882e1d13038fc48ba2602af9de4dd2e # bad: [a70a5c33a65ee54048e4ae479e3479d765a1bbc2] Merge branch into tip/mas= ter: 'core/core' git bisect bad a70a5c33a65ee54048e4ae479e3479d765a1bbc2 # good: [85e511df3cec46021024176672a748008ed135bf] sched/eevdf: Allow short= er slices to wakeup-preempt git bisect good 85e511df3cec46021024176672a748008ed135bf # good: [20f13385b5836d00d64698748565fc6d3ce9b419] posix-timers: Consolidat= e timer setup git bisect good 20f13385b5836d00d64698748565fc6d3ce9b419 # good: [42db2c2cb5ac3572380a9489b8f8bbe0e534dfc7] timekeeping: Use time_af= ter() in timekeeping_check_update() git bisect good 42db2c2cb5ac3572380a9489b8f8bbe0e534dfc7 # bad: [3a0c7230588b40caf1d81270ceaa3aa5c0355bc0] Merge branch into tip/mas= ter: 'perf/urgent' git bisect bad 3a0c7230588b40caf1d81270ceaa3aa5c0355bc0 # bad: [de752774f38bb766941ed1bf910ba5a9f6cc6bf7] jump_label: Fix static_ke= y_slow_dec() yet again git bisect bad de752774f38bb766941ed1bf910ba5a9f6cc6bf7 # good: [fe513c2ef0a172a58f158e2e70465c4317f0a9a2] static_call: Replace poi= ntless WARN_ON() in static_call_module_notify() git bisect good fe513c2ef0a172a58f158e2e70465c4317f0a9a2 # first bad commit: [de752774f38bb766941ed1bf910ba5a9f6cc6bf7] jump_label: = Fix static_key_slow_dec() yet again --NYw/P0L4bhK7Sn5+ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAABCgAdFiEEreZoqmdXGLWf4p/qJNaLcl1Uh9AFAmbgqRMACgkQJNaLcl1U h9CaoAf/WEHszCEtu0E/ZLuteJCN/8fHtjcQYf85VbcmRn4ar2Tx/s0lQycNbup+ vYz/MmdyatYIOkepv6Bog0K9z0vCYNgTssvide+cfoNBwz8wB9/c4+HaUFhznNoH XO9Tx/ImFpMEefF3lv3QwVKyGAZHzjsGaN/wMm9HktMnSW9VkW6QcbwuG04Aq2mP jBk1Mh0PZ5jl0aE2cS1+Z/l8UiUY5q8XSiuUoD8qdb4U9iOG/K24ZFIFVusfm5b5 /HyxU4+bSybwLxz61+MOB+r8yJYgKnToEwbXt2nu2lQikns9WHDAWjGBTnbnxvhX uVKTIB3QrlzkMDiEwhWIVD41mKLecw== =6Vrc -----END PGP SIGNATURE----- --NYw/P0L4bhK7Sn5+--