From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazon11011070.outbound.protection.outlook.com [40.93.194.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A8103D9023 for ; Tue, 31 Mar 2026 10:19:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.194.70 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774952354; cv=fail; b=BZGS5/h2nhbtn0j6vy/X0UM3wjn7hhYML0IEJXcHn0OQv5eYNj3QPOBLbCFO0I0s7DnANbIesoDNPSSJFVpYUq4cbyor6TTIIfa2frNSKfEB277fVmQXwKzOkMuQAgiudSfKeEgxfhevFTwynYqgicVb8UEnUtyYSdjS9jWyrDs= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774952354; c=relaxed/simple; bh=TgAYR9Utz9f/TaocZJ5jjM7lUQ/akFKpmsfnRM+nKCg=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=mvguoofwB0boMiLNM84jchSotyOt1HYcUiWB+PENs5xA7g4eaOVOOVDLNwE0R0Qtcey/Q7W+8hwFwdH5b0gURNC21pHapvxnS2wAURkTVuPCJEr/hkas1YkCB2uVWeXSsBo0VvldMOyIkvahU9j4lG8wwi+AQmSEPe7DeKcYwsM= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=vsH2Hdno; arc=fail smtp.client-ip=40.93.194.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="vsH2Hdno" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QK/uE921q9BLif6+32z2TYo6T/9yHA2s+HqhiWY2SAFrTD2wjVZvjJqfXbVXUTGZFitM19vttIfgmwgzcjs6KDwrOZooDsr5A19Xsd5w7i4r8nX/TpATsqizSxfyzaWMVxqJ+c0qgFQ0ZTx3rqakruQ1kJ7rKePuqX8JZuddufGsIsDegbyNmuTVSd48aivKM83MXgSOWmFrlD/02Ef3u4rBTWhbkosg6kWf9XpC7jjWjNO47+rP8+1uzGCnyZQLG5TmFEU2q9Md3f3Gs/JwbvY5VfjPkSJsnyJ/k7Gs7EvX70WQAouqXIf9klXTGLNLPAmDHowtzwxN94ZWVoUCIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jU26+vJ0LEflrz6FmqULm8rudy4TOPva7twXlhoi8qE=; b=U0cWvAJAgtzxMWneCeduEmUFKF8hWFy78WSL7FoJkOMmB765gd+ugN4WKi3ZGKGeb13xssEf8W5ZoS2VWbZJWZGdmAELlrqvlW6PEao/MkfslGuZxUTE9QeTQA6yeAuuMxTW4WUwqO4woXRtfOMJQW4QWAoFH0EVTz7c6GA9Vnquvscs53WOO0Yxe0v7F/t6W8UqXZHq151AN9nVXulU7Kny4w+dxJ1StGIWYsbMa/hWGUmoe/g69k7dO+ecO4RJYSfzqQxp99y9Ca5j/yL3WZ6LwWpU+UCqpkpzzT/PAZ0JJ+bFqxznjIfIlEuHNybaHIAcsVnSasc1bw9qTAj5Vg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=intel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jU26+vJ0LEflrz6FmqULm8rudy4TOPva7twXlhoi8qE=; b=vsH2HdnovNa6qqKm7m0Ru5iAA+ooR8jFMLYEdzbpxCHwGg59WPvICj5JkyR9hA4KLC74e2tDKTaCZ2APnwuDSq6kWm5UmjziZP9K5Z2xh70G/XfoAqof6AW7rhpIsyh+dic/LfojoOEThx2Ip9lljv9uoBlhx3wZSMa4ZcGC3Ds= Received: from DM6PR06CA0046.namprd06.prod.outlook.com (2603:10b6:5:54::23) by PH7PR12MB7161.namprd12.prod.outlook.com (2603:10b6:510:200::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.14; Tue, 31 Mar 2026 10:19:09 +0000 Received: from DM2PEPF00003FC5.namprd04.prod.outlook.com (2603:10b6:5:54:cafe::bb) by DM6PR06CA0046.outlook.office365.com (2603:10b6:5:54::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9745.28 via Frontend Transport; Tue, 31 Mar 2026 10:19:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by DM2PEPF00003FC5.mail.protection.outlook.com (10.167.23.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9745.21 via Frontend Transport; Tue, 31 Mar 2026 10:19:08 +0000 Received: from satlexmb10.amd.com (10.181.42.219) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 31 Mar 2026 05:19:08 -0500 Received: from satlexmb07.amd.com (10.181.42.216) by satlexmb10.amd.com (10.181.42.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 31 Mar 2026 05:19:07 -0500 Received: from [10.136.44.157] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Tue, 31 Mar 2026 05:19:06 -0500 Message-ID: <22072ef8-5aec-49ac-9cc4-8a80bec14261@amd.com> Date: Tue, 31 Mar 2026 15:49:05 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 4/4] sched/rt: Split cpupri_vec->cpumask to per NUMA node to reduce contention To: "Chen, Yu C" , Peter Zijlstra , Tim Chen CC: Pan Deng , , , References: <20260320124003.GU3738786@noisy.programming.kicks-ass.net> <63a095f02428700a7ff2623b8ea81e524a406834.camel@linux.intel.com> <20260324120008.GB3738010@noisy.programming.kicks-ass.net> <138c3f9d-309f-41e6-aa72-a3f6bd713bf0@intel.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <138c3f9d-309f-41e6-aa72-a3f6bd713bf0@intel.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM2PEPF00003FC5:EE_|PH7PR12MB7161:EE_ X-MS-Office365-Filtering-Correlation-Id: e9988d0f-d6a4-45fc-0f0d-08de8f0ef219 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|30052699003|36860700016|376014|22082099003|56012099003|18002099003; X-Microsoft-Antispam-Message-Info: mUUFmKTOo0K0h6qE8DQCBq3C5gOQNS31YFzOWCGF3aPUsltX0fIkTzTL6Yi7vDAMgZMQGMep+MSn5K9OPoexs2v0tcabC4wkls3Fnntx6YoRgowHwlr0wDMPl1bQJk0GQtlOeTNSKc6/3rkPf0Jz2dhmHe08rT8Zuz3NbgtOqnDelbyztrmRoWyX0Rf+qA7N3F9PHD81AJT+XLamElX5u2bRUH8kv1HRAc40GgebURXur/otC5Q7AjnQiGoq4gC/RB97W2WbZvefB98j3EFuvtmNbmHILtxc73CKMGyOnQQoIJAqXz4r2ZeiE1b3P0JrhhS98D7KjclOSHEKt32+7yOUeptV7X+RU+Kc/g92NqMnscI64/rCCkPQ3A+c0Flb32bepNs9d1bxT2Ryxu+FhfakhVX8kzZmf0GHknadMuGDw4jZ01MPwJBYe2XaCcfT5N9IAIxTFZLJnaSL68S7juCoS6rCXSISxAOEvN9PhMC/24CSOx5i2OEjYtb5K4ztg7As17aX0gGqkcpuOXO7+Guj+agTss0Kwl9MTgqBXCb85o9j1jHj9n6qVmnfVuN7FikRDrw9AxbGcTTR7jCl/3L/CHXTuqxWpClNAhaHEcFB+6PSBrt87I08KZhJjKZOo9ZoLoEO9fGow3ehQ9PvtUMPWpS5nQUShv0kW85KXQORUzytXv7LZbhYHv+P5+1K9A3qpyS99LLH9xHLDfc4ZO4ID9cb8mtUU1se8mQjoMAK64jwkAeLriSyoQ6tg3J7a5MBq92tH123iZX3VXMYsQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(30052699003)(36860700016)(376014)(22082099003)(56012099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: WoF8USCRJuMfu3Szwd9ujkx8ypnwXfMnUzyZu4Ixg3Hn03AWCGx6iWic93HTVPYRwnf46tsQcvmxI0doEF63aA0t8TVcgylCqQvkJ4zPPHQMlkdUOxhSuaH78GxWWrAWf65frj/VZkzIQwLbzP/oteC2Abqg0HybZWOJdy1zsX3ASr7ZOyyozJhm57AzCxZuQxV5sRBdZqj2xssL7y8c2e1W/me60gC5LLge4kXEsgrj2dGvadK6ZfdtYLdXkRC7A94/DZ/5o4yReCy0altQutFJ2EDkhFHZCO+nOL/IitRp5psq1wDFbHsVMXjFy0Z6jP6tMuN4d50TXc4BphsDAaLjhSwHqFTKE9gWnerOWkw9i1l+X8h2IjNM35g59nqMiq2lom+X4utKrhumXIogSVUDLsyCUiWKio8gpEcDn1kN7B4x23vtG4Z/6IZ5WwmJ X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Mar 2026 10:19:08.6425 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e9988d0f-d6a4-45fc-0f0d-08de8f0ef219 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM2PEPF00003FC5.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7161 Hello Chenyu, On 3/31/2026 11:07 AM, Chen, Yu C wrote: > update of the test: > With above change, I did a simple hackbench test on > a system with multiple LLCs within 1 node, so the benefit > is significant(+12%~+30%) when system is under-loaded, while > some regression when overloaded(-10%)(need to figure out) Could it be because of how we are traversing the CPUs now for idle load balancing? Since we use the first set bit for ilb_cpu and also staring balancing from that very CPu, we might just stop after a successful balance on the ilb_cpu. Would something like below on top of Peter's suggestion + your fix help? (lightly tested; Has survived sched messaging on baremetal) diff --git a/include/linux/sbm.h b/include/linux/sbm.h index 8beade6c0585..98c4c1866534 100644 --- a/include/linux/sbm.h +++ b/include/linux/sbm.h @@ -76,8 +76,45 @@ static inline bool sbm_cpu_test(struct sbm *sbm, int cpu) return __sbm_op(sbm, test_bit); } +static __always_inline +unsigned int sbm_find_next_bit_wrap(struct sbm *sbm, int start) +{ + int bit = sbm_find_next_bit(sbm, start); + + if (bit >= 0 || start == 0) + return bit; + + bit = sbm_find_next_bit(sbm, 0); + return bit < start ? bit : -1; +} + +static __always_inline +unsigned int __sbm_for_each_wrap(struct sbm *sbm, int start, int n) +{ + int bit; + + /* If not wrapped around */ + if (n > start) { + /* and have a bit, just return it. */ + bit = sbm_find_next_bit(sbm, n); + if (bit >= 0) + return bit; + + /* Otherwise, wrap around and ... */ + n = 0; + } + + /* Search the other part. */ + bit = sbm_find_next_bit(sbm, n); + return bit < start ? bit : -1; +} + #define sbm_for_each_set_bit(sbm, idx) \ for (int idx = sbm_find_next_bit(sbm, 0); \ idx >= 0; idx = sbm_find_next_bit(sbm, idx+1)) +#define sbm_for_each_set_bit_wrap(sbm, idx, start) \ + for (int idx = sbm_find_next_bit_wrap(sbm, start); \ + idx >= 0; idx = __sbm_for_each_wrap(sbm, start, idx+1)) + #endif /* _LINUX_SBM_H */ diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a3a423c4706e..f485afb6286d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12916,6 +12916,7 @@ static void _nohz_idle_balance(struct rq *this_rq, unsigned int flags) int this_cpu = this_rq->cpu; int balance_cpu; struct rq *rq; + u32 start; WARN_ON_ONCE((flags & NOHZ_KICK_MASK) == NOHZ_BALANCE_KICK); @@ -12944,7 +12945,8 @@ static void _nohz_idle_balance(struct rq *this_rq, unsigned int flags) * Start with the next CPU after this_cpu so we will end with this_cpu and let a * chance for other idle cpu to pull load. */ - sbm_for_each_set_bit(nohz.sbm, idx) { + start = arch_sbm_cpu_to_idx((this_cpu + 1) % nr_cpu_ids); + sbm_for_each_set_bit_wrap(nohz.sbm, idx, start) { balance_cpu = arch_sbm_idx_to_cpu(idx); if (!idle_cpu(balance_cpu)) --- This is pretty much giving me similar performance as tip for sched messaging runs under heavy load but your mileage may vary :-) -- Thanks and Regards, Prateek