From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F26F3191BB for ; Tue, 4 Aug 2026 12:37:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785847050; cv=none; b=Afu6XYU1Mg3TVHK9hCTIiIVXdEMC02WtcKEVSVlqXp9FJFA4DGvk/d4HDWuYIWtTsS2sz3JVSnMimBekRv80s56HPE2dkHwBGvTbTOe5Pa2fFMz+ofT+6tXl60rBoSn1aMPRJR9q5voQM7T7q8sJ05WBMVl7N4HPhlwiELJAVBc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785847050; c=relaxed/simple; bh=3RbtBE3YjMQ/Fwaakkr0a/AxNrjP7n/ILdTRno1jS9M=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=oEdHPxNP5UthpoEcOonfrJULeTSHRc+gDEgAtyDrZeilApDOScdaXBOQWIeX3aZOMS6ULBkdja90IfmeByk3yQjzEIEHijVWlirluXYOw4BTFSJp17IUhIMY/vEre5wTXPuz3iC0YYxAgE1+AnFF00y8DiGaT+BaBwj0CM1ZfaU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=AXjc4jxN; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="AXjc4jxN" Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 6748I7CA324383; Tue, 4 Aug 2026 12:36:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=g4pvO8 VSyBpaiQmkUJPRjj2k09/O6nuxftz7k1zuqc0=; b=AXjc4jxNRpH4NdZ36r1jMo ZL3NofIdPsobNuA7BhJB0ho5YbrEoE7JxOSFC0hB7D2TAz76ngPq8ukuKUFzUlE7 dn8T+MisOzR5cY2isZpI3VsyxvrcDvVdF/o0GR2/WhTszcHadSZtUdEvAFpBLAKE jRGBhwepZvN1lFaCwlq+yT2nZtSYxvQxPv93V1GITTjh1TXHFWQDeAKNnOJMUfWm +1SAeo7aZkMorlhOEru3AoXwNiCYn0pwTvY7fHMBlPNDYX3OmcPSNi8Y6cCKXeOr JXrMc0+wTZx1Iki5v2R+7a/Zy89oVSOf9HsmT41FMou6AB7kVppnS4t2bCADryHw == Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fs8h4wtja-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 04 Aug 2026 12:36:52 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 674CQW0l001478; Tue, 4 Aug 2026 12:36:51 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fsv4k1wng-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 04 Aug 2026 12:36:51 +0000 (GMT) Received: from smtpav03.fra02v.mail.ibm.com (smtpav03.fra02v.mail.ibm.com [10.20.54.102]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 674CanUm40632788 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 4 Aug 2026 12:36:49 GMT Received: from smtpav03.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 321042004E; Tue, 4 Aug 2026 12:36:49 +0000 (GMT) Received: from smtpav03.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CB0A120043; Tue, 4 Aug 2026 12:36:48 +0000 (GMT) Received: from [9.224.76.67] (unknown [9.224.76.67]) by smtpav03.fra02v.mail.ibm.com (Postfix) with ESMTP; Tue, 4 Aug 2026 12:36:48 +0000 (GMT) Message-ID: <533a1615-e0f8-4c7d-b212-b24760f198d3@linux.ibm.com> Date: Tue, 4 Aug 2026 14:36:48 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing To: Andrea Righi , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Shrikanth Hegde , Phil Auld , linux-kernel@vger.kernel.org References: <20260731191957.3199642-1-arighi@nvidia.com> Content-Language: en-US From: Mete Durlu In-Reply-To: <20260731191957.3199642-1-arighi@nvidia.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Proofpoint-Spam-Info: AW1haW4tMjYwODA0MDA5OCBTYWx0ZWRfX34H+c77CFV8V Dnlf2gM2CQ2nzHWMXwj8zuj6GMVQh3nD/k1ouVOVYEC0PCpMrp/mbivKJIyIAQr5J/yWvglfRjx c4Z9IFEmZ4bOSGAlo9BlfM7uoYGKha0= X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwODA0MDA5OCBTYWx0ZWRfX2qZFzY1oCfx+ WnsjIPnSWGY5c1eTypt20tD1gxj50+A7QVSImeiTq1nv/pM893Cubdy7oleDskg3t8N7Vk5SSz1 UBcCPrlKDNhMVgUV+PL9OflG2npwWvpY9/0VrSx0SvCwnzPX0tjULkaZ+MR5cyWCj4YXxRwqDlt TD0ra/wKfU0Dl+R6kHMRK36CydcC7r3B89RKcDs9ma6AgvHZ72OQ91ClnG+cs7Zge+ohilk0ok2 gNFn4l1/Bvhky3wLQBpkLFA7bi27pUoY8/01ZyjZ3xolqIgVCKFcUOW58Tj9uTyhYP+8innjCNG QRqr0+gLqho+m+OcxyeCgx6T0JTUJMkNClR0W3bgR8NxdIuTgvtIulYmvhH91qhdcAmBcxhFypi KEXeY17i9irOLFIv01P/cJTLGrofct6beJZU1tGlRTiKMFakVu0sE3ZF9X5wHQ4l0N14Tn8iF9V kCzm3EDOj2+PsCY5swQ== X-Authority-Analysis: v=2.4 cv=SI1ykuvH c=1 sm=1 tr=0 ts=6a71dce5 cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=IkcTkHD0fZMA:10 a=Sv0fKeRqtYgA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=U7nrCbtTmkRpXpFmAIza:22 a=VnNF1IyMAAAA:8 a=tc9LG2_lRxzPEkKOlpcA:9 a=QEXdDO2ut3YA:10 X-Proofpoint-ORIG-GUID: ZPsCCUSFTFdFnPjQCI4FvNmKp7KGD9Zg X-Proofpoint-GUID: o1rKXqIip0eNszBN-XuPqp6ya9aZF1u4 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1176,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-08-04_02,2026-08-03_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 bulkscore=0 suspectscore=0 impostorscore=0 spamscore=0 phishscore=0 priorityscore=1501 lowpriorityscore=0 adultscore=0 malwarescore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2608040098 Hi, > find_new_ilb() selects the first idle housekeeping CPU without > considering whether another thread is running on the same physical core. > On an SMT system, the idle load balancer can therefore activate both > siblings even when another housekeeping CPU has an entirely idle core. > > On most SMT systems, this is not problematic because the idle load > balancer is a short-lived activity and the transient wakeup of a sibling > has negligible performance impact. > > However, this can be particularly costly on NVIDIA Olympus cores used in > Vera. Briefly activating an otherwise idle sibling can reduce the > performance available to the other sibling and this effect does not > necessarily end once the activated sibling becomes idle: after the ILB > finishes and its CPU enters WFI, full single-thread performance is > restored only after the sibling has remained idle for a qualification > interval (10 Ki cycles on the tested Vera system). Repeated short > sibling wakeups can therefore sustain the interference even with little > actual overlap. > > Prevent this by preferring an idle housekeeping CPU whose entire SMT > core is idle. Retain the first idle CPU as a fallback when no fully idle > core is available, so NOHZ balancing continues to make forward progress. > Once a partially busy core has been examined, skip its remaining SMT > siblings to avoid repeating the core-idle check on wide SMT systems. > > Tests performed using an ad hoc GEMM benchmark running one CPU-intensive > task per SMT core within its CPU affinity mask improved from > approximately 6.2 TFLOP/s to 9.4 TFLOP/s. Although what you describe above with siblings suffering interference does not really fit to s390, I'd like to hear more about what sort of GEMM (general matrix multiplication) tests you did. I tested this patch with a couple of different tools - perf bench sched pipe - hackbench - uperf - cyclictest - stress-ng (3d-matrix and cyclic) Didn't come across any meaningful difference in any of them on multiple runs each. So I was curious about the exact sort of benchmark you mention here. One minor nit for the diff below; > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 37001c63452e5..574b6b3ee922a 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq) > static inline int find_new_ilb(void) > { > int this_cpu = smp_processor_id(); > - const struct cpumask *hk_mask; > - int ilb_cpu; > + struct cpumask *ilb_cpus; > + int ilb_cpu, fallback = -1; > + > + lockdep_assert_irqs_disabled(); > > - hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE); > + /* > + * Reuse the per-CPU select_rq_mask, which is protected from concurrent > + * use on this CPU by having interrupts disabled. > + */ > + ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask); > + cpumask_and(ilb_cpus, nohz.idle_cpus_mask, > + housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)); > > - for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) { > + for_each_cpu(ilb_cpu, ilb_cpus) { > if (ilb_cpu == this_cpu) > continue; > > - if (idle_cpu(ilb_cpu)) > - return ilb_cpu; > + if (!idle_cpu(ilb_cpu)) { > + /* > + * Once an idle fallback exists, a busy CPU proves that > + * this core cannot be fully idle. Skip its siblings. > + */ > + if (sched_smt_active() && fallback >= 0) > + cpumask_andnot(ilb_cpus, ilb_cpus, > + cpu_smt_mask(ilb_cpu)); nit; With line break this if block is now taking multiple lines and deserves its own curly braces. With or without the nit, feel free to add my r-b to v4, I doubt removal of the "this_cpu" check will change anything as it is a dud. Reviewed By: Mete Durlu