From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5290BC43332 for ; Wed, 6 Jan 2021 04:50:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2002E22BF3 for ; Wed, 6 Jan 2021 04:50:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727189AbhAFEuy (ORCPT ); Tue, 5 Jan 2021 23:50:54 -0500 Received: from mail.kernel.org ([198.145.29.99]:33428 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726929AbhAFEuS (ORCPT ); Tue, 5 Jan 2021 23:50:18 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 874B823100; Wed, 6 Jan 2021 04:48:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1609908537; bh=Qzy/YwWUsOGCBsgvf0qpdXbbOiGRjfOUm9RbKUJgSO0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J1BSEDmTn+w0DOq/0xetVNLHYC3UBqkbOJvgdYsv0Pdaj8I8MwWlv1pNqW4+f6hEl WzjRI7ZW7pJDMlUzjJgjZdhmEnT9HM4aaV6JbAWfcxh7SMvyzhYM6YZ5a4zTSqGZVN 9TCkHqQ5A2cnQD1Cktj4FXrhURaiexSwZx2U2nZA7nrRA4S4zgrOcM2+RXfZ8u66Ov io3Jb/C0ygkhsMO5yXa3f1Q6fcA2Gq4jmvgNi19+3dULURosodZJ9QLzx8tOw8slEZ aD6bXcc6UXDS2OlyvAScpIOuoSWP/mdngL/02RfIUg84W7v6/1rqN+9pyYgs2KZOwA Iyzu0ke/trzIQ== From: paulmck@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, mingo@kernel.org, jiangshanlai@gmail.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, joel@joelfernandes.org, Frederic Weisbecker , Neeraj Upadhyay , "Paul E . McKenney" Subject: [PATCH tip/core/rcu 13/21] rcu/nocb: Locally accelerate callbacks as long as offloading isn't complete Date: Tue, 5 Jan 2021 20:48:45 -0800 Message-Id: <20210106044853.20812-13-paulmck@kernel.org> X-Mailer: git-send-email 2.9.5 In-Reply-To: <20210106013950.GA14663@paulmck-ThinkPad-P72> References: <20210106013950.GA14663@paulmck-ThinkPad-P72> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Frederic Weisbecker The local callbacks processing checks if any callbacks need acceleration. This commit carries out this checking under nocb lock protection in the middle of toggle operations, during which time rcu_core() executes concurrently with GP/CB kthreads. Cc: Josh Triplett Cc: Steven Rostedt Cc: Mathieu Desnoyers Cc: Lai Jiangshan Cc: Joel Fernandes Cc: Neeraj Upadhyay Cc: Thomas Gleixner Inspired-by: Paul E. McKenney Tested-by: Boqun Feng Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index ec14c01..03810a5 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2699,7 +2699,6 @@ static __latent_entropy void rcu_core(void) unsigned long flags; struct rcu_data *rdp = raw_cpu_ptr(&rcu_data); struct rcu_node *rnp = rdp->mynode; - const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist); const bool do_batch = !rcu_segcblist_completely_offloaded(&rdp->cblist); if (cpu_is_offline(smp_processor_id())) @@ -2720,11 +2719,11 @@ static __latent_entropy void rcu_core(void) /* No grace period and unregistered callbacks? */ if (!rcu_gp_in_progress() && - rcu_segcblist_is_enabled(&rdp->cblist) && !offloaded) { - local_irq_save(flags); + rcu_segcblist_is_enabled(&rdp->cblist) && do_batch) { + rcu_nocb_lock_irqsave(rdp, flags); if (!rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL)) rcu_accelerate_cbs_unlocked(rnp, rdp); - local_irq_restore(flags); + rcu_nocb_unlock_irqrestore(rdp, flags); } rcu_check_gp_start_stall(rnp, rdp, rcu_jiffies_till_stall_check()); -- 2.9.5