From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11021091.outbound.protection.outlook.com [52.101.62.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2AA941760 for ; Thu, 12 Feb 2026 20:04:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.91 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770926661; cv=fail; b=ICEYBHivdyaiP3iNO/PQB8lX42strgZ2UJ/0i7UPHOmFFxzcNrAD3sWQeaJ2OQt4Q3ils7hiE/dbHP/ExdqV15GK+8niXbGfzW2MQsZK2ja88E9GbMH4FOOoKUz4ktaFuka/12O8VB6gXc9EqYggfHok2sUAdVuqKDp0XaBqNXY= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770926661; c=relaxed/simple; bh=BVUpZ/UfWeiEJj2q/Lig0YeiGgULDp5BAgvsjYpfbCI=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: Content-Type:MIME-Version; b=nntO8yMwWZq+bEAM7mRn+pI3bWERgcIBfUyUIrrof9F9jeJFcXLUrmhWgOae/rDDQm0no27evWr5FXppgJWjx3kwa52xfeenrlvJGR3ppU8YdumUGZJ0+kO1GCvmCZckU+r9YSxd7lbsd8DcXHXaRUwMlahpMh7C4u0alixgGME= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=D73UMJNk; arc=fail smtp.client-ip=52.101.62.91 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="D73UMJNk" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=r2SOd2qcL1Mw4pTfHEuNXVNgeehFS1C/t3QuGrE1YEFNJBV819PG6IKQk22hF7a7fmWrlwwPWlk3RXWIiUJg1uL/++aSk4wqFsBbABESW9cZ/sngIaOXKQ8NhJmH4DyLJx4nuwcFbfq3eO/qz7H9oeqtjvO/nwevddh7Iq0651gn6tGTYZmCU/5m8lIpZmxkeZcG6BXOC4z6Y0j5UwTyOrsb17vg0n6FBcRpNyEZiuVxcynJW3HTZLJySSuF+EGfMkIPezkJAeTJdRWDODTEBo8ijS++KNFo27n368kCWbSnfN2u8UU9yQ8ewrJ5LB7ZnBYycF/c977C7peHs7aDIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cj/Gy7HKYjLwqPmmXmZkO6TEf6qmnqDgAE1ds2qq9+o=; b=RY5w5IDa9Vi/JQanpxj44mH1IqWRFQOdsd1VSb2lw0qQGu9l8slsOtUAaYhJOaK/y+9Be2AbGWbLeUklOXZNulRJublrZWYJ2VFilP5yv9jHMTN73epRaiu8Cfef0oeHJcjd+TGVd8tMqLQ8rPLGnRfM8ULvnV5dRbydBv4bqQeeWkAOMLto/Jvetg3Va67Xq/ucVni4Op0bcInJTyblcAgOmqdWk24vWpp8LyM7aHNV/iSRlgjcAB6yhNhBPkVV+s0wcm14YMav/QAam3ccmt4xHAuwhfE7QsJqX9Q9yqBx9WqpVuMHCgZMG3ReAFW6/ahhe7bEW9ckV5SVcCFcbg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cj/Gy7HKYjLwqPmmXmZkO6TEf6qmnqDgAE1ds2qq9+o=; b=D73UMJNkwIXsp8Zw3jyBS+t8Lv1du6kRfVIWyeu8v2AwkLQE97v0wuFX6SzBJQbAznW4Q5ZxGyb3nqWhLExy4ZhskDw88H606hhWRjqenRwC5NFz1MWic7walitjUYtE5uJREqa713bIfJVUOLqdIaa3+n7PDQ4Iu1WPTCm4gyI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from DM2PR01MB9464.prod.exchangelabs.com (2603:10b6:8:2e4::17) by CO1PR01MB6789.prod.exchangelabs.com (2603:10b6:303:fa::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9611.10; Thu, 12 Feb 2026 20:04:16 +0000 Received: from DM2PR01MB9464.prod.exchangelabs.com ([fe80::8b41:b64c:97d4:ed48]) by DM2PR01MB9464.prod.exchangelabs.com ([fe80::8b41:b64c:97d4:ed48%4]) with mapi id 15.20.9611.008; Thu, 12 Feb 2026 20:04:14 +0000 Date: Thu, 12 Feb 2026 12:04:11 -0800 (PST) From: Shubhang Kaushik To: Frederic Weisbecker cc: Anna-Maria Behnsen , Ingo Molnar , Thomas Gleixner , Vincent Guittot , Valentin Schneider , dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, rostedt@goodmis.org, Christoph Lameter , linux-kernel@vger.kernel.org, Adam Li Subject: Re: [RESEND PATCH] tick/nohz: Fix wrong NOHZ idle CPU state In-Reply-To: Message-ID: References: <20260203-fix-nohz-idle-v1-1-ad05a5872080@os.amperecomputing.com> Content-Type: text/plain; charset=US-ASCII; format=flowed X-ClientProxiedBy: CYXPR02CA0057.namprd02.prod.outlook.com (2603:10b6:930:cd::15) To DM2PR01MB9464.prod.exchangelabs.com (2603:10b6:8:2e4::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM2PR01MB9464:EE_|CO1PR01MB6789:EE_ X-MS-Office365-Filtering-Correlation-Id: 37f44bea-7c67-40e7-2e10-08de6a71e57e X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?WSyxjs3/TWm2c/U8/2bXVQ7dVDpDJbQzI2bJrNgw0PS6dw29KtTzT96HXohp?= =?us-ascii?Q?BG/lYT3t1bxpRSOBbbrpFnnSVYOo+rYs/M6RgYZrekfaCCCQugXUW38IMuX7?= =?us-ascii?Q?mE0T120J71tvTTagiOJNQF020mElzD5ksSPCrokK5U0k+ocTFCCOfeJEclDL?= =?us-ascii?Q?pgEr1jX5z/GFDIJHdR1oJ2B5cHVDuvBEXOMoE2D2WE2mCEbREBWHxvxbz6kE?= =?us-ascii?Q?w7ejK5OYV2gdQlqrnTFWVyfvlEIUq7LmNNpgXLyyxlSdEpFBtl+4piGM25KV?= =?us-ascii?Q?cKMl0lt/dc8yy4yk+3VSowL9HoiFbU+PkB4lYyiSkctCwkVh5UjKpGPstirK?= =?us-ascii?Q?seMLAnPYmII/eAQz1dPu5AsHT0p+nsqYANxWZYRGhx5XtT9DYoCzBGB3zInn?= =?us-ascii?Q?plpvyPNqoHEYhDeKljdIyg78Msl7ewQ86NUUB1uJvjjO0N3qJzokWkDWiz/Q?= =?us-ascii?Q?eq+GAF0u75IpYAzpOdFOcBXfHom12RrvPmz5I9ZlPv4tOH+F4iGE3b8yZIyB?= =?us-ascii?Q?I9ixFlj/ahA16oBes9aexUt0DmvguGnd6UzAUWIujzlhw06D8T7HVbWiik/W?= =?us-ascii?Q?uSm3Z1ncaunEbt4MyAMRPDacue+3gZU/F+M//rp1p2VV5cjBBbE/vor3semG?= =?us-ascii?Q?UnS/7R10RFzpPuXECo1mxZBjAHcUDM5pKH1zfeMTTuahbV5JSzNvSZDGKEqh?= =?us-ascii?Q?kpfVFDIh0tft7ujWwL5XjjoJq+NxRIgEuAIMfzgRtXFAIttMXejKCo9esx2m?= =?us-ascii?Q?tMWJp+/i0OW0Bdf+nzFyjefFzHcn/2hsKwFUpQ8wnPKQAMjEIlPDe6+mWg4X?= =?us-ascii?Q?g5ArZJVsygyJ7MWeaexQLCowBQCDgoqpm5KzX4b1zzC2a5RZnMpdw82cOu1t?= =?us-ascii?Q?ruK0+wRwaSqhuYbX+TdS5Wi65prcyCdqVYAGFQ5AepB08GCpbB3nPqkai1sT?= =?us-ascii?Q?sJ5lf+caMXaKt+F4c1SEIASPtyIeXXU5rbERI1/tAkMPhrFuG+2fC8iXBlaJ?= =?us-ascii?Q?/bIxP7rgNrU3XRX1s+6ma4AyEsSuKWYaRfLcPiDZFsIbdGotSWdWYeMmOS7X?= =?us-ascii?Q?/aNpqQg+IKCQypcYKUUb2kjIwHnWyUIWwXbPrdSWkmKIU00dQ3aI7KsNHGVH?= =?us-ascii?Q?Sj4I/yXRnwNv8rtyrUJLOgsanWHRV1JlqOWp/0YHGTn7cSYqJF4desmiyS7A?= =?us-ascii?Q?6tXdsKdwXrUpk7RXLbQSnPvFTHwJp5j+O5DZa6Fs0ScCJ0aKV2BuLDtLRF+I?= =?us-ascii?Q?+LVh7Ai8XcPhqx2ZBJkfATDE0345x3xsUDmZMLbdU24zv3sER4sPXCxbo75T?= =?us-ascii?Q?9RisRdvijmAXvj8HWnbEmaw/jtPoNcsLID6l9MZOYEM4wjby8xgmm7cNxHIa?= =?us-ascii?Q?j1V0ukcZ9t69aVOryUhU+QKvmXipCb1kPb31LuhIlgqH1OoIHlAj9CCG8TES?= =?us-ascii?Q?baf0o3xbWfQZ9wtcY/3ZigmTXYXtApz1n3V6L9Wp87Dn9kRgF26e7c2NGTsr?= =?us-ascii?Q?4wdZ8QUja1Qeu1LTFQPJpC1lUfnM61n8uU9bxr+SfXnELn3KCGi9BFLxQRan?= =?us-ascii?Q?0JE98gHqsIxlHya97Wc=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM2PR01MB9464.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(7416014)(376014)(1800799024);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7IrSkRYPtWZe7fnSfbrWCbrm865ssi2/KThl0aQFNbMtyc8tIRJgrCBBepGI?= =?us-ascii?Q?+5PMbOuCVibnDLO2RMNGBzGoGCuPq8sRJs3B98tUbLaGwDUiGFsBQ2zDoaWa?= =?us-ascii?Q?34zrn0VXDMaavBj24fLJagZ4m3eaHjPeWOeq8xfqCgfz+QwbCoN9MJK40d6X?= =?us-ascii?Q?GKFgYzD1sNIWGPHr2RJ7ZXmQL92UCCKzn8ZkjI3XtIA8WYG3Or2a8TdlD877?= =?us-ascii?Q?tfJ/LzchFjnVb5KiUiasSvy2PLuWDeXarhZ4Si4sLyq7OTh1EtpWDXEDnfNO?= =?us-ascii?Q?qPiCSYsSUFNYEfvcx5IsUUyQfFDRyHxd8g9OKBR2CIGM6tRNihggZyauWqza?= =?us-ascii?Q?TjIwvzVM1OpJL3hQ1feWIrPcJdR+Nva9W8+n2bChn+p8xQ4Hza7gmQdh+Qvj?= =?us-ascii?Q?87u05VByBzSKkROw01s9Bd1Z0L7lrB5ziDe7wdP5fxVtQJmjYEFseHN6TF9B?= =?us-ascii?Q?WSoDsZ/mKXS++9fT8TSFuOpgoleauIdeJ5BbDoUefwrRPSzc2La5Rn7CGvKA?= =?us-ascii?Q?KWo1ZZ3cnCdyvnfXc+RJxbD8LXmZ7OSt0NSxL+DlBE6bPCqP9WIG5BoQ9q9K?= =?us-ascii?Q?hD8xDoghf7r4wKd/9et1JbX8a50yKprPCZrx2I4NiDfS6960d211wBgndulw?= =?us-ascii?Q?xQhGaPmyPcH4cDni8D4nfK+DkybTndmd68dmitetUggtk5fslgMgrLPIDOem?= =?us-ascii?Q?xOcNfjnuoEjGJp+xEwqpZhirJ1hg7qBZCi+Li9zkwFyGnHZnKDipvUkJejK5?= =?us-ascii?Q?7VHngrRjhXsCgX0YCj5Bc6x4hhpa/O6XkR6rqZZgf4/Rm/MzSqf7z23dCe68?= =?us-ascii?Q?h/idhMG7R6caq2uamM4l6dgTEfTVTkFMbOQpj2MUt5T7xZE0UJ8oI9zIEDpa?= =?us-ascii?Q?ztcDQiRlT7+VQrOtOyuiXGIuVuhmmr2nS2wLxyu3zG6DzAcsOv2j1BrfS+ap?= =?us-ascii?Q?MXKCaoFrwAr8/4P4QDQMhh5R0zHMvSqGaZUYc9htf6MlOUCMpH7ZGKn7XI8q?= =?us-ascii?Q?7S0cOzcJWBwHmshFpLnPWCC+oDdGRVVW7W57BXyeq53oaSySKzGZV0rK34su?= =?us-ascii?Q?YgGAX41tey40CdNAELRfof1mtMFuno+2jAFabTwgg4FDOFX8JwJGlxYNIf0v?= =?us-ascii?Q?jFe9OODVpUAdK0dBmovMldMUqrLDX97RZgXFR5dnl5WigCxIb9Sp6rp9Jy9T?= =?us-ascii?Q?f6JEDXaSYyn3AjHnv0oPMgg2Sh0URgUPCKKSIhQL95MSsCsXkQx6OqqBrmLJ?= =?us-ascii?Q?gtrSDPbbedyISnbMuVus4vwjQR8cZhprarT9TmFjOTZcYyIfK9L3gGaBARIv?= =?us-ascii?Q?OfnnCgjK5XtB2GhuaeuxNdpNHYyVSeEEKmjtqqID11knRMdJkwMmi5BmlCAH?= =?us-ascii?Q?VW3Hwqs7DIlpwsRieRVcJtHzBawCV0cv7PkwFWpuW/Z/od6nfEMr32uz5Dcn?= =?us-ascii?Q?VUlqxVSDoOJwGt4QWpBM4gUmFobawqAHSJZHIpkMpgKA5bSHne5mYO2OKsJk?= =?us-ascii?Q?o0LJ7JTVgr3pylk9caoOofV6htP77+Q3DtrC0NJms/JEbmySNjr3YETkbMfh?= =?us-ascii?Q?9BhaEf8dz9Oqz3eUfTSXXwFuDjRfascsTf6AVrAMcv8on/DZYwP6ThrZbuo9?= =?us-ascii?Q?QeJT1fqa3gZ3OmPiDtPAGm8jIbhsXhvTp4oPwiV4DeMWZGXnY8+qyi1Yh/nl?= =?us-ascii?Q?KSsnJ2AegU7abPq7laGE0mLS9g1Y0fuA/SCL8iYNatN410slcpXmelbfj8po?= =?us-ascii?Q?jBwfdVQrL9MtP7UZkDO2xSbI6R2aFsuD6GjvMHnhVd3vdx3ruMOM?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 37f44bea-7c67-40e7-2e10-08de6a71e57e X-MS-Exchange-CrossTenant-AuthSource: DM2PR01MB9464.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Feb 2026 20:04:14.9166 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9AwX9qqbf3UzfZged/ttZPd0gujPy0ehNBRAU6N6kEbmPM74hGV/kjiJKtB3g55ivu5lviCPHnf6SeuiGigIrWBH6tqjPyOVw1suAt+LUE2aCtNl0Fw/cQuRr8ZiyHR7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR01MB6789 On Thu, 12 Feb 2026, Shubhang Kaushik wrote: >> Because you rely on dynamic placement of isolated tasks throughout >> isolated >> CPUs by the scheduler. >> >> But nohz_full is designed for running only one task per isolated CPU >> without >> any disturbance. And migration is a significant disturbance. This is why >> nohz_full tries not to be too smart and assumes that task placement is >> entirely >> within the hands of the user. >> >> So I have to ask, what prevents you from using static task placement in >> your >> workload? > > Actually, the llama-batched-bench results I shared already included static > affinity testing via numactl -C. What I mean by that is even when tasks are strictly pinned to individual cores, the performance gap remains. IIUC, the current implementation assumes tick-stop and idle-entry are coupled. While this holds for standard NOHZ, nohz_full decouples them, causing idle CPUs to be omitted from nohz.idle_cpus_mask. This hides idle capacity from the NOHZ idle balancer, forcing housekeeping tasks onto active cores. By decoupling these transitions in the code, we ensure accurate state accounting. > > Even with static placement, we observe this ~14% throughput improvement. This > suggests that the issue isn't about the scheduler trying to be smart with > task migration, but rather about the side effects of an idle CPU being absent > from nohz.idle_cpus_mask. > > When nohz_full CPUs enter idle but aren't correctly accounted for in the idle > mask, it appears to cause unnecessary overhead or interference in the NOHZ > load balancing logic for the CPUs that are still running tasks. By ensuring > the idle state is correctly tracked, we're not encouraging migration, but > rather ensuring the scheduler's global state accurately reflects reality. > > AFAICT this seems to be a case where correcting the bookkeeping benefits HPC > throughput even when the user handles all task placement manually. > > Regards, > Shubhang Kaushik >> >> I'm not saying it's undesirable or impossible to do adaptive userspace >> dyntick >> for users that don't rely on ultra low latency but rather on high >> CPU-bound >> performance. In fact the initial purpose of nohz_full was for HPC and not >> real-time. Turns out that real time is all the usecase I have seen so far >> and >> you're the first HPC one. But adapting nohz_full dynamically for that will >> involve >> much more than just load balancing. Now the static affinity should work >> for >> everyone. >> >> Thanks. >> >> >>> >>> Signed-off-by: Shubhang Kaushik >>> Signed-off-by: Adam Li >>> Reviewed-by: Christoph Lameter (Ampere) >>> Reviewed-by: Shubhang Kaushik >>> --- >>> This is a resend of the original patch to ensure visibility. >>> Previous resend: https://lkml.org/lkml/2025/8/21/170 >>> Original thread: https://lkml.org/lkml/2025/8/21/171 >>> >>> The patch addresses a performance regression in NOHZ idle load balancing >>> observed under CONFIG_NO_HZ_FULL, where idle CPUs were becoming >>> invisible to the balancer. >>> --- >>> kernel/time/tick-sched.c | 5 +++-- >>> 1 file changed, 3 insertions(+), 2 deletions(-) >>> >>> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c >>> index >>> 2f8a7923fa279409ffe950f770ff2eac868f6ece..eee6fcebe78c2f8d93464a55fe332e12fe9c164e >>> 100644 >>> --- a/kernel/time/tick-sched.c >>> +++ b/kernel/time/tick-sched.c >>> @@ -1250,8 +1250,9 @@ void tick_nohz_idle_stop_tick(void) >>> ts->idle_sleeps++; >>> ts->idle_expires = expires; >>> >>> - if (!was_stopped && tick_sched_flag_test(ts, >>> TS_FLAG_STOPPED)) { >>> - ts->idle_jiffies = ts->last_jiffies; >>> + if (tick_sched_flag_test(ts, TS_FLAG_STOPPED)) { >>> + if (!was_stopped) >>> + ts->idle_jiffies = ts->last_jiffies; >>> nohz_balance_enter_idle(cpu); >>> } >>> } else { >>> >>> --- >>> base-commit: 18f7fcd5e69a04df57b563360b88be72471d6b62 >>> change-id: 20260203-fix-nohz-idle-b2838276cb91 >>> >>> Best regards, >>> -- >>> Shubhang Kaushik >>> >> >> -- >> Frederic Weisbecker >> SUSE Labs >> > >