From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5D1AC41535 for ; Tue, 10 May 2022 14:18:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344447AbiEJOWG (ORCPT ); Tue, 10 May 2022 10:22:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245002AbiEJNrL (ORCPT ); Tue, 10 May 2022 09:47:11 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 072053B3CF; Tue, 10 May 2022 06:33:30 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B0F57B81D24; Tue, 10 May 2022 13:33:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BA72EC385C2; Tue, 10 May 2022 13:33:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1652189607; bh=irB8NTev9xa9lpY3OQpWk1fLdyWlxyfi8AVBIBkf/Pk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jw97UV4SHoLcX5ChbJvMpjlNlbN97ZoksGsvFZqqS7uLD5FVOMcSpW7gFa8rm2nDC WDBMQ+/adYk5gfMmGR4mHfYKWJRvBf/6EOaWDMwzsCbe9e4Bqj0ZtxPfNd71CWkCMg ZGeRkr5slDi2oS5v/EIpMUal6WdXnFnT/1SZWjJU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Valentin Schneider , Sebastian Andrzej Siewior , Frederic Weisbecker , Peter Zijlstra , Josh Triplett , Joel Fernandes , Boqun Feng , Neeraj Upadhyay , Uladzislau Rezki , Thomas Gleixner , "Paul E. McKenney" Subject: [PATCH 5.15 105/135] rcu: Apply callbacks processing time limit only on softirq Date: Tue, 10 May 2022 15:08:07 +0200 Message-Id: <20220510130743.419301937@linuxfoundation.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220510130740.392653815@linuxfoundation.org> References: <20220510130740.392653815@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Frederic Weisbecker commit a554ba288845fd3f6f12311fd76a51694233458a upstream. Time limit only makes sense when callbacks are serviced in softirq mode because: _ In case we need to get back to the scheduler, cond_resched_tasks_rcu_qs() is called after each callback. _ In case some other softirq vector needs the CPU, the call to local_bh_enable() before cond_resched_tasks_rcu_qs() takes care about them via a call to do_softirq(). Therefore, make sure the time limit only applies to softirq mode. Reviewed-by: Valentin Schneider Tested-by: Valentin Schneider Tested-by: Sebastian Andrzej Siewior Signed-off-by: Frederic Weisbecker Cc: Valentin Schneider Cc: Peter Zijlstra Cc: Sebastian Andrzej Siewior Cc: Josh Triplett Cc: Joel Fernandes Cc: Boqun Feng Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Thomas Gleixner Signed-off-by: Paul E. McKenney [UR: backport to 5.15-stable] Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Greg Kroah-Hartman --- kernel/rcu/tree.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2476,7 +2476,7 @@ static void rcu_do_batch(struct rcu_data div = READ_ONCE(rcu_divisor); div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div; bl = max(rdp->blimit, pending >> div); - if (unlikely(bl > 100)) { + if (in_serving_softirq() && unlikely(bl > 100)) { long rrn = READ_ONCE(rcu_resched_ns); rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn; @@ -2517,6 +2517,18 @@ static void rcu_do_batch(struct rcu_data if (count >= bl && (need_resched() || (!is_idle_task(current) && !rcu_is_callbacks_kthread()))) break; + + /* + * Make sure we don't spend too much time here and deprive other + * softirq vectors of CPU cycles. + */ + if (unlikely(tlimit)) { + /* only call local_clock() every 32 callbacks */ + if (likely((count & 31) || local_clock() < tlimit)) + continue; + /* Exceeded the time limit, so leave. */ + break; + } } else { local_bh_enable(); lockdep_assert_irqs_enabled(); @@ -2524,18 +2536,6 @@ static void rcu_do_batch(struct rcu_data lockdep_assert_irqs_enabled(); local_bh_disable(); } - - /* - * Make sure we don't spend too much time here and deprive other - * softirq vectors of CPU cycles. - */ - if (unlikely(tlimit)) { - /* only call local_clock() every 32 callbacks */ - if (likely((count & 31) || local_clock() < tlimit)) - continue; - /* Exceeded the time limit, so leave. */ - break; - } } local_irq_save(flags);