From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76501C433EF for ; Tue, 19 Oct 2021 00:08:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 60CA961212 for ; Tue, 19 Oct 2021 00:08:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233840AbhJSALI (ORCPT ); Mon, 18 Oct 2021 20:11:08 -0400 Received: from mail.kernel.org ([198.145.29.99]:50782 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234155AbhJSALB (ORCPT ); Mon, 18 Oct 2021 20:11:01 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id B411761359; Tue, 19 Oct 2021 00:08:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634602129; bh=X0lyfZZ5ZrgLTH98qOWagyvT/lUxKIn7Sn0KGPVpcNc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ecn7/vdio52GsQrlb9P8j3sNz3UWkIdsPXnKAIqKXttttEuEVdLTLrh3F7dBHJbYc h0Z0fbpTQqwkY3enbC8J4PmDY9ROvcvY3iwASQ2zd33pMlKC6MG1Hg3OSNwgBMCofD OCsWErhrgr4ugpQMs+VeLrFzwgTOBREtp1zDSw8fyGrRiw26TAB0gx/mfoeocc4Zed gPjCV0EEBbZD9LdKMRrhTiFf8iarnYRweXXBiqJstmQ9I2Vh3j/86Mo5351IGK5cYR dBKx3h5weDxgfljdisPQvkpQvs8fWbw/855NS969D7U5iDpiKv8r9CJ5zxPCvOQkL4 46kXzLLgON8FA== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Sebastian Andrzej Siewior , Valentin Schneider , Peter Zijlstra , Uladzislau Rezki , Thomas Gleixner , Valentin Schneider , Boqun Feng , Neeraj Upadhyay , Josh Triplett , Joel Fernandes Subject: [PATCH 09/10] rcu: Apply callbacks processing time limit only on softirq Date: Tue, 19 Oct 2021 02:08:15 +0200 Message-Id: <20211019000816.455375-10-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211019000816.455375-1-frederic@kernel.org> References: <20211019000816.455375-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Time limit only makes sense when callbacks are serviced in softirq mode because: _ In case we need to get back to the scheduler, cond_resched_tasks_rcu_qs() is called after each callback. _ In case some other softirq vector needs the CPU, the call to local_bh_enable() before cond_resched_tasks_rcu_qs() takes care about them via a call to do_softirq(). Therefore, make sure the time limit only applies to softirq mode. Reviewed-by: Valentin Schneider Tested-by: Valentin Schneider Tested-by: Sebastian Andrzej Siewior Signed-off-by: Frederic Weisbecker Cc: Valentin Schneider Cc: Peter Zijlstra Cc: Sebastian Andrzej Siewior Cc: Josh Triplett Cc: Joel Fernandes Cc: Boqun Feng Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Thomas Gleixner Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7a655d93a28a..0ca4f4ec724d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2498,7 +2498,7 @@ static void rcu_do_batch(struct rcu_data *rdp) div = READ_ONCE(rcu_divisor); div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div; bl = max(rdp->blimit, pending >> div); - if (unlikely(bl > 100)) { + if (in_serving_softirq() && unlikely(bl > 100)) { long rrn = READ_ONCE(rcu_resched_ns); rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn; @@ -2538,6 +2538,17 @@ static void rcu_do_batch(struct rcu_data *rdp) if (in_serving_softirq()) { if (count >= bl && (need_resched() || !is_idle_task(current))) break; + /* + * Make sure we don't spend too much time here and deprive other + * softirq vectors of CPU cycles. + */ + if (unlikely(tlimit)) { + /* only call local_clock() every 32 callbacks */ + if (likely((count & 31) || local_clock() < tlimit)) + continue; + /* Exceeded the time limit, so leave. */ + break; + } } else { local_bh_enable(); lockdep_assert_irqs_enabled(); @@ -2545,18 +2556,6 @@ static void rcu_do_batch(struct rcu_data *rdp) lockdep_assert_irqs_enabled(); local_bh_disable(); } - - /* - * Make sure we don't spend too much time here and deprive other - * softirq vectors of CPU cycles. - */ - if (unlikely(tlimit)) { - /* only call local_clock() every 32 callbacks */ - if (likely((count & 31) || local_clock() < tlimit)) - continue; - /* Exceeded the time limit, so leave. */ - break; - } } rcu_nocb_lock_irqsave(rdp, flags); -- 2.25.1