From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACDBBC64ED6 for ; Sun, 26 Feb 2023 15:58:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229608AbjBZP6v (ORCPT ); Sun, 26 Feb 2023 10:58:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229804AbjBZP6k (ORCPT ); Sun, 26 Feb 2023 10:58:40 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3393A86BE; Sun, 26 Feb 2023 07:58:38 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 676B1B80BEC; Sun, 26 Feb 2023 14:49:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1336AC4339B; Sun, 26 Feb 2023 14:48:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1677422939; bh=X3M7PUXFrQ6Z26Iu4BXLC0L/s/3ZL7m3kT+bp3ozSzo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gW9Wfyx3zMnYuoP7WJu56q3YGF3f5N9d8lHQjoUFidNIeBAJD+EDAgWZtuoTqjrMu +2YLXJvt+rjWafhj3O5LKexPFrp0Vv09FeUGoKRUVnIgJ4C2jk2WN6LZ/HdalITBKN p7dqbzHy7/PBhHSmUlIVH7KPe++7ZJZePRdvrfD6XSYdw6oajrxDlnsPrEpL5lEwYD 57XAm+3nX1tBRgLCEGLY7ac259LkMUlHggcdyO7Q1dpBhIaHGvqMVUUPp/7GUj89HO hvC++lkr1AzEMsXrScrVlto58lMvKcZ9Gk7i+/dCVFx8z0+zsg3ziJ+N4TGKvXwhwB vci0P32ibIqDQ== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Zqiang , "Paul E . McKenney" , Sasha Levin , frederic@kernel.org, quic_neeraju@quicinc.com, josh@joshtriplett.org, rcu@vger.kernel.org Subject: [PATCH AUTOSEL 5.15 05/36] rcu-tasks: Make rude RCU-Tasks work well with CPU hotplug Date: Sun, 26 Feb 2023 09:48:13 -0500 Message-Id: <20230226144845.827893-5-sashal@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230226144845.827893-1-sashal@kernel.org> References: <20230226144845.827893-1-sashal@kernel.org> MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zqiang [ Upstream commit ea5c8987fef20a8cca07e428aa28bc64649c5104 ] The synchronize_rcu_tasks_rude() function invokes rcu_tasks_rude_wait_gp() to wait one rude RCU-tasks grace period. The rcu_tasks_rude_wait_gp() function in turn checks if there is only a single online CPU. If so, it will immediately return, because a call to synchronize_rcu_tasks_rude() is by definition a grace period on a single-CPU system. (We could have blocked!) Unfortunately, this check uses num_online_cpus() without synchronization, which can result in too-short grace periods. To see this, consider the following scenario: CPU0 CPU1 (going offline) migration/1 task: cpu_stopper_thread -> take_cpu_down -> _cpu_disable (dec __num_online_cpus) ->cpuhp_invoke_callback preempt_disable access old_data0 task1 del old_data0 ..... synchronize_rcu_tasks_rude() task1 schedule out .... task2 schedule in rcu_tasks_rude_wait_gp() ->__num_online_cpus == 1 ->return .... task1 schedule in ->free old_data0 preempt_enable When CPU1 decrements __num_online_cpus, its value becomes 1. However, CPU1 has not finished going offline, and will take one last trip through the scheduler and the idle loop before it actually stops executing instructions. Because synchronize_rcu_tasks_rude() is mostly used for tracing, and because both the scheduler and the idle loop can be traced, this means that CPU0's prematurely ended grace period might disrupt the tracing on CPU1. Given that this disruption might include CPU1 executing instructions in memory that was just now freed (and maybe reallocated), this is a matter of some concern. This commit therefore removes that problematic single-CPU check from the rcu_tasks_rude_wait_gp() function. This dispenses with the single-CPU optimization, but there is no evidence indicating that this optimization is important. In addition, synchronize_rcu_tasks_generic() contains a similar optimization (albeit only for early boot), which also splats. (As in exactly why are you invoking synchronize_rcu_tasks_rude() so early in boot, anyway???) It is OK for the synchronize_rcu_tasks_rude() function's check to be unsynchronized because the only times that this check can evaluate to true is when there is only a single CPU running with preemption disabled. While in the area, this commit also fixes a minor bug in which a call to synchronize_rcu_tasks_rude() would instead be attributed to synchronize_rcu_tasks(). [ paulmck: Add "synchronize_" prefix and "()" suffix. ] Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney Signed-off-by: Sasha Levin --- kernel/rcu/tasks.h | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 4bd07cc3c0eab..1f6a8ec7704a4 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -171,8 +171,9 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func, static void synchronize_rcu_tasks_generic(struct rcu_tasks *rtp) { /* Complain if the scheduler has not started. */ - WARN_ONCE(rcu_scheduler_active == RCU_SCHEDULER_INACTIVE, - "synchronize_rcu_tasks called too soon"); + if (WARN_ONCE(rcu_scheduler_active == RCU_SCHEDULER_INACTIVE, + "synchronize_%s() called too soon", rtp->name)) + return; /* Wait for the grace period. */ wait_rcu_gp(rtp->call_func); @@ -660,9 +661,6 @@ static void rcu_tasks_be_rude(struct work_struct *work) // Wait for one rude RCU-tasks grace period. static void rcu_tasks_rude_wait_gp(struct rcu_tasks *rtp) { - if (num_online_cpus() <= 1) - return; // Fastpath for only one CPU. - rtp->n_ipis += cpumask_weight(cpu_online_mask); schedule_on_each_cpu(rcu_tasks_be_rude); } -- 2.39.0