From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D6CC47CC86 for ; Mon, 11 May 2026 17:55:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778522102; cv=none; b=R+9Shy1zoB/Vj8iMufvx5efF62wJz2g3lQMyMm7EK6iqcjRxcTqxbBdQVHphGa/kt/vMlobaP7PL3p/o8i/E4eXthhndxktkfl+bf8USaCtqtQ2PRo2zibVdf5LDN+Z+wFxpj9ngFpE8Nvx2Rby+ICU9suuTc5wYw7Ke6+STxzc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778522102; c=relaxed/simple; bh=N1ZRO99Xmu7eMJ4ncdKQ4oBgaMAOlGsvAg+Q2j1/DMo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AfTCNRg9DZyoW4l+QksN/29j9E57jQQUEgsnmfvjIgXS0Lns5qhbOlbsiNg97tStvbExyy9vkSReyYGezoZaSAMx3MqV5Iky2I3ynhAXZ2XFRYN2oBX1ItKN0eoWYGQ3pss1Ikpq4Ww5N646JvZDt23SEGXT0sH9bJpQC6KW3Oc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=HZ7cMTqx; arc=none smtp.client-ip=209.85.208.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HZ7cMTqx" Received: by mail-lj1-f179.google.com with SMTP id 38308e7fff4ca-393da8f389bso45939521fa.1 for ; Mon, 11 May 2026 10:55:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1778522098; x=1779126898; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OSjCy54KooYgtOnKS1P4NfKbs3McoPoMDdY9GznQwyk=; b=HZ7cMTqxfT4oI8TfOkrV2pTIkLTUmrbXVDkOZjCXxPU2nSKLkuO6UyaP1XfRceUUf/ 452fjdrXJ6ys2hTuHXWuiRxbSCvUGzrLayG+SDPFPp3PFm+z159Yl3y8Zb53GkQ/bzMd IGVQJKvx5OsKVaX8PF/L0ja/h0Jndln9cs+gpI5EL/qNha1qQQMi2tb/pWeE8mJALTXp yCEsKxnCiPcactCBNfgrx2Cxt+N/e1fQJ8bX2wxpi9iCbVVg7MTadkjcG/GoaDvKDqFD 2QoJaRNg+dhMNVZeq+L6GzD350PZJtuQ+0Uo7auvxqcBuNHYyaYhsA16ikFRqAKClDE8 UQkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778522098; x=1779126898; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OSjCy54KooYgtOnKS1P4NfKbs3McoPoMDdY9GznQwyk=; b=a9k4b90Up2akw9oDMHWQNy+zy3FMC/rlbJXG4l8Tslx6ZKg++8qJp/oSSayI/JwAWq UNVV5zD0UWtQ2BNLLqACrcxHAJUCFGroEwSd6m5dHREZxaUcpkl4qv74qMkdp2o+cu7Z rL9NpTzATDod8qiXYP5EfkuUsx7w3g/g4HBA1CVEedWpouBjq89EvK4dzri3Cx3Owbcy SMpLDrXT5JkIV3oMhP8ZcszsAaIJnBCJInHcNzjHJVrRNCQLdkp6pQfIQmM4rek0w6v6 BxoWNcAHskNJ2Nvyr5A/9xvcpcy2Y29INnCdFbVkRqb1L9iCt8+slNhQ1Ohadkjc9fCQ zpZg== X-Forwarded-Encrypted: i=1; AFNElJ8wkVrcK0EuP766xTUlm86a6HnNVqtIGuvdDtLDWr/XvAigkBwRMLWdTfjd/DaJAhMtUl8uGkHkuTGHdkM=@vger.kernel.org X-Gm-Message-State: AOJu0Yw9+ea96lIJzWqCIyhrSxb43hP8HppHHgL6LTzCfBFS6qhCpbBX GNyan4seUEbxc74xLRYPat0m60UzTfw3sBOIMCUfKu8/tcrQ4fpZQsMK X-Gm-Gg: Acq92OE6kAJNb9JKmKqZ30aCaUbISWpIRacgCmHvS3sdUZc2u3rwMW12340msKl1Svg tyrspoFQsoXxrm6P3yA/lZFpJLxdqqxbBBOieponr8/LO0JeI05JQM5CFvkzzVRoBrG8xlT8k6D k95US3B3mk46GcCV8q3gnj9VkyCm6rUuLMUi1nTGOLocjthhE6XOujqBmEL9WD0mW6s503jtq/Y 0jsNa2QPGtSy3eqphAZVIaug+MMN/KxKk+7zGDWwYJn9a5IDtmheI5sWVPpXRUEy3baRMZIqXuf jtC2KZpYpyXPTILswDqzOvy8BvxKqgPO6gD3C5L/xlJZCP9HbHfXxeNbh0/D0f8PSYrHOPzVVxJ 9y3v0GMcxQjcpzBOUbGuLxBp92YAL8UVRIKeitykr+TqMcF8fIM93sUCk3Hn2cu32vNR+Adwmgq +ielexKLaFKeRrqjo= X-Received: by 2002:a2e:a596:0:b0:392:5c61:c583 with SMTP id 38308e7fff4ca-39408157c68mr38748381fa.23.1778522098290; Mon, 11 May 2026 10:54:58 -0700 (PDT) Received: from localhost.localdomain ([2001:9b1:d5a0:a500::24b]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-393f614a5f4sm28968891fa.33.2026.05.11.10.54.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2026 10:54:57 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" , Joel Fernandes , Frederic Weisbecker , Boqun Feng Cc: RCU , LKML , Uladzislau Rezki , Samir M Subject: [PATCH -next v1 11/12] rcu: Latch normal synchronize_rcu() path on flood Date: Mon, 11 May 2026 19:54:47 +0200 Message-ID: <20260511175448.153326-12-urezki@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260511175448.153326-1-urezki@gmail.com> References: <20260511175448.153326-1-urezki@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Currently, rcu_normal_wake_from_gp is only enabled by default on small systems(<= 16 CPUs) or when a user explicitly set it enabled. Introduce an adaptive latching mechanism: * Track the number of in-flight synchronize_rcu() requests using a new rcu_sr_normal_count counter; * If the count reaches/exceeds RCU_SR_NORMAL_LATCH_THR(64), it sets the rcu_sr_normal_latched, reverting new requests onto the scaled wait_rcu_gp() path; * The latch is cleared only when the pending requests are fully drained(nr == 0); * Enables rcu_normal_wake_from_gp by default for all systems, relying on this dynamic throttling instead of static CPU limits. Testing(synthetic flood workload): * Kernel version: 6.19.0-rc6 * Number of CPUs: 1536 * 60K concurrent synchronize_rcu() calls Perf(cycles, system-wide): total cycles: 932020263832 rcu_sr_normal_add_req(): 2650282811 cycles(~0.28%) Perf report excerpt: 0.01% 0.01% sync_test/... [k] rcu_sr_normal_add_req Measured overhead of rcu_sr_normal_add_req() remained ~0.28% of total CPU cycles in this synthetic stress test. Tested-by: Samir M Suggested-by: Joel Fernandes Signed-off-by: Uladzislau Rezki (Sony) --- .../admin-guide/kernel-parameters.txt | 10 ++-- kernel/rcu/tree.c | 52 ++++++++++++++----- 2 files changed, 44 insertions(+), 18 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 4d0f545fb3ec..d5db2e85d551 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5862,13 +5862,13 @@ Kernel parameters use a call_rcu[_hurry]() path. Please note, this is for a normal grace period. - How to enable it: + How to disable it: - echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp - or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1" + echo 0 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp + or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=0" - Default is 1 if num_possible_cpus() <= 16 and it is not explicitly - disabled by the boot parameter passing 0. + Default is 1 if it is not explicitly disabled by the boot parameter + passing 0. rcuscale.gp_async= [KNL] Measure performance of asynchronous diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 09f0cef5014c..94274330d1db 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1632,17 +1632,21 @@ static void rcu_sr_put_wait_head(struct llist_node *node) atomic_set_release(&sr_wn->inuse, 0); } -/* Enable rcu_normal_wake_from_gp automatically on small systems. */ -#define WAKE_FROM_GP_CPU_THRESHOLD 16 - -static int rcu_normal_wake_from_gp = -1; +static int rcu_normal_wake_from_gp = 1; module_param(rcu_normal_wake_from_gp, int, 0644); static struct workqueue_struct *sync_wq; +#define RCU_SR_NORMAL_LATCH_THR 64 + +/* Number of in-flight synchronize_rcu() calls queued on srs_next. */ +static atomic_long_t rcu_sr_normal_count; +static int rcu_sr_normal_latched; /* 0/1 */ + static void rcu_sr_normal_complete(struct llist_node *node) { struct rcu_synchronize *rs = container_of( (struct rcu_head *) node, struct rcu_synchronize, head); + long nr; WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && !poll_state_synchronize_rcu_full(&rs->oldstate), @@ -1650,6 +1654,15 @@ static void rcu_sr_normal_complete(struct llist_node *node) /* Finally. */ complete(&rs->completion); + nr = atomic_long_dec_return(&rcu_sr_normal_count); + WARN_ON_ONCE(nr < 0); + + /* + * Unlatch: switch back to normal path when fully + * drained and if it has been latched. + */ + if (nr == 0) + (void)cmpxchg(&rcu_sr_normal_latched, 1, 0); } static void rcu_sr_normal_gp_cleanup_work(struct work_struct *work) @@ -1795,6 +1808,24 @@ static bool rcu_sr_normal_gp_init(void) static void rcu_sr_normal_add_req(struct rcu_synchronize *rs) { + /* + * Increment before publish to avoid a complete + * vs enqueue race on latch. + */ + long nr = atomic_long_inc_return(&rcu_sr_normal_count); + + /* + * Latch when threshold is reached. Checking for an exact match + * restricts cmpxchg() to a single context. + * + * This latch is intentionally relaxed and best-effort. Concurrent + * set/clear can race and temporarily lose the latch, which is OK + * because it only selects between the fast and fallback paths. + */ + if (nr == RCU_SR_NORMAL_LATCH_THR) + (void)cmpxchg(&rcu_sr_normal_latched, 0, 1); + + /* Publish for the GP kthread/worker. */ llist_add((struct llist_node *) &rs->head, &rcu_state.srs_next); } @@ -3278,14 +3309,15 @@ static void synchronize_rcu_normal(void) { struct rcu_synchronize rs; + init_rcu_head_on_stack(&rs.head); trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request")); - if (READ_ONCE(rcu_normal_wake_from_gp) < 1) { + if (READ_ONCE(rcu_normal_wake_from_gp) < 1 || + READ_ONCE(rcu_sr_normal_latched)) { wait_rcu_gp(call_rcu_hurry); goto trace_complete_out; } - init_rcu_head_on_stack(&rs.head); init_completion(&rs.completion); /* @@ -3302,10 +3334,10 @@ static void synchronize_rcu_normal(void) /* Now we can wait. */ wait_for_completion(&rs.completion); - destroy_rcu_head_on_stack(&rs.head); trace_complete_out: trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("complete")); + destroy_rcu_head_on_stack(&rs.head); } /** @@ -4904,12 +4936,6 @@ void __init rcu_init(void) sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM | WQ_UNBOUND, 0); WARN_ON(!sync_wq); - /* Respect if explicitly disabled via a boot parameter. */ - if (rcu_normal_wake_from_gp < 0) { - if (num_possible_cpus() <= WAKE_FROM_GP_CPU_THRESHOLD) - rcu_normal_wake_from_gp = 1; - } - /* Fill in default value for rcutree.qovld boot parameter. */ /* -After- the rcu_node ->lock fields are initialized! */ if (qovld < 0) -- 2.47.3