From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-002e3701.pphosted.com (mx0b-002e3701.pphosted.com [148.163.143.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 107AA329C49 for ; Mon, 23 Feb 2026 17:12:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.143.35 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771866762; cv=none; b=U9nhVT5j8FZNo30pwXhz538G/hjFEhQiAdmEBlzt86U3+7swbkbmA50eXGGCfzYSrA/E13CLTp2McCuxMelVnAMlJbdhRIXXyJzYTFeHCNfcguQYv/9tRlSEV942gJ9orHCDraPTlZk1t2g0CxaYi9vOspeA6wesf6TD43kR7oA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771866762; c=relaxed/simple; bh=nv8ary0qwttsGfycNnKAU6JOU7BdEgj9wWfmfqHc9kw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=qqOAHEwmb9ZmwLvOGyUrGMU0tdrL0gpBEMAFznJ67/VR2hLm0DXG+BOlWX0svHrgLvDE9/EC75VbzzyccRWI/tu9ZZkggKHjV+QyT48Iy9qQMB+wfXk4wUGPfx0fb17yWdxPr0idZWQLE5S4rJqlzxPCUH5DP6tzqmWdPW8jXjQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=hpe.com; spf=pass smtp.mailfrom=hpe.com; dkim=pass (2048-bit key) header.d=hpe.com header.i=@hpe.com header.b=P0KcBWzU; arc=none smtp.client-ip=148.163.143.35 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=hpe.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=hpe.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hpe.com header.i=@hpe.com header.b="P0KcBWzU" Received: from pps.filterd (m0148664.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 61NEqUvL1554786; Mon, 23 Feb 2026 16:42:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=cc :content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to; s=pps0720; bh=YX40DN7PjpYjPcXuqFAs2vt0x4 +o+r2ybqFyZc2QVZA=; b=P0KcBWzUae6WQZ2lzSVwKD3aJA5/GajT3yyDkuAVsX cfESsHRYZ92iVxs/w+5kmhJcao5iBbsT7SDCP7lcwOPw01CKuFn5JsRXQNztDg7u ayi2Gkf0YrRyedxtcj4KmWbknv0jWGgzhC80w9Dh1QELzeMWO6jH6xDihoaBIKwz irBxTj1SKpWOdCdMpE4t6YTxdX39dDMxUo0iqSFSAPPQsOOg1i30sVUfFBveHDKF IrWJgrLeeDQj+3sZnaiWRoVpgirJ91oLZn9fFKGMkZynnFtXzD4Hm5GWEHaQlP5Y JRGzvny9SGFvVJG7Y1MfvdTTWhiu1bSTB4O4qlTztdJA== Received: from p1lg14881.it.hpe.com (p1lg14881.it.hpe.com [16.230.97.202]) by mx0b-002e3701.pphosted.com (PPS) with ESMTPS id 4cgs1mhd30-1 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Mon, 23 Feb 2026 16:42:43 +0000 (GMT) Received: from p1lg14886.dc01.its.hpecorp.net (unknown [10.119.18.237]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14881.it.hpe.com (Postfix) with ESMTPS id BEA6A80479E; Mon, 23 Feb 2026 16:42:38 +0000 (UTC) Received: from HPE-5CG20646DK.localdomain (unknown [16.231.227.36]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by p1lg14886.dc01.its.hpecorp.net (Postfix) with ESMTPS id 2602E816DB7; Mon, 23 Feb 2026 16:42:36 +0000 (UTC) Date: Mon, 23 Feb 2026 10:42:34 -0600 From: Kyle Meyer To: tim.c.chen@linux.intel.com, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, peterz@infradead.org, tglx@kernel.org, vinicius.gomes@intel.com Cc: brgerst@gmail.com, hpa@zytor.com, kprateek.nayak@amd.com, linux-kernel@vger.kernel.org, patryk.wlazlyn@linux.intel.com, rafael.j.wysocki@intel.com, russ.anderson@hpe.com, x86@kernel.org, yu.c.chen@intel.com, zhao1.liu@intel.com Subject: Re: [PATCH v2] sched/topology: Check average distances to remote packages Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Proofpoint-GUID: 3obaQzTRLCqvZ_Iow4vAkdmjavsXLOk- X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwMjIzMDE0MyBTYWx0ZWRfX56EekiMgmwwO qXybk806waGE5W82BM07FCWl1Wtta5Ad2w+2ETWWgo+7fqfay9E6xB80078MIHs+/FkCYfXlxVm +E2XHZIoY6uBQ+A0w+UJYBneodPGj0eJ+2XDTqrERWSyOb6fmx9OPVe0vOvXFmN/h8GWD6xMQQf Vinf8NE8yVuDFib7Dqy/n97b1E7tcFQ742p22FCjzNgbubCRI/iHBGPi3X2Vu9SmhgCOHkuUamx /0bEZ/4pIoTQ/CB57shV0AyVJE43EUd6FA1mluL9G56HNN2q7kJO1Dp9tsFTi87itnjplA1oE+E bPB7Apau0jfTT+dLhgJhNEsAU59YCv0cG+PkS8m/qIj3E3tq18TEdNWrkmbIvzD4pl7pA1V9wsl 8bLuEEKlbKnVVkiy2UAmk2wgYUklsSYetBUPOG6eevOauAqFfXl8LnphZzFLzE36SqLeg46P8a2 8ijt5KQxNqIQqS0OYbQ== X-Proofpoint-ORIG-GUID: 3obaQzTRLCqvZ_Iow4vAkdmjavsXLOk- X-Authority-Analysis: v=2.4 cv=Wuwm8Nfv c=1 sm=1 tr=0 ts=699c8384 cx=c_pps a=FAnPgvRYq/vnBSvlTDCQOQ==:117 a=FAnPgvRYq/vnBSvlTDCQOQ==:17 a=kj9zAlcOel0A:10 a=HzLeVaNsDn8A:10 a=VkNPw1HP01LnGYTKEx00:22 a=Mpw57Om8IfrbqaoTuvik:22 a=GgsMoib0sEa3-_RKJdDe:22 a=VwQbUJbxAAAA:8 a=MvuuwTCpAAAA:8 a=QyXUC8HyAAAA:8 a=FbW2QPigx-Tb4Mov8U0A:9 a=CjuIK1q_8ugA:10 X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1121,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-02-23_03,2026-02-23_03,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 lowpriorityscore=0 adultscore=0 priorityscore=1501 phishscore=0 bulkscore=0 suspectscore=0 impostorscore=0 malwarescore=0 clxscore=1015 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2601150000 definitions=main-2602230143 On Wed, Feb 04, 2026 at 06:24:31PM -0600, Kyle Meyer wrote: > Granite Rapids (GNR) and Clearwater Forest (CWF) average distances to > remote packages to fix scheduler domains, see [1] for more information. > > A warning and backtrace are printed when sub-NUMA clustering (SNC) is > enabled and there are more than 2 packages because the average distances > to remote packages could be different, skewing the single average remote > distance. > > This is unnecessary when the average distances to remote packages are > the same. > > Support single average remote distance on systems with more than 2 > packages, preventing unnecessary warnings and backtraces, by checking if > average distances to remote packages are the same. > > [1] commit 4d6dd05d07d0 ("sched/topology: Fix sched domain build error for GNR, CWF in SNC-3 mode"). > > Reviewed-by: Tim Chen > Signed-off-by: Kyle Meyer > --- > > The warning and backtrace were noticed on a 16 socket GNR system with SNC-2 enabled. > > v1: > * https://lore.kernel.org/all/aXjvLjTCRe8d3UFD@hpe.com/ > > v1 -> v2: > * Initialize pkg_total_distance and pkg_nr_remote to NULL, as suggested by Tim. > > --- > arch/x86/kernel/smpboot.c | 69 ++++++++++++++++++++++++++++----------- > 1 file changed, 50 insertions(+), 19 deletions(-) > > diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c > index 5cd6950ab672..dc8f15bd2e19 100644 > --- a/arch/x86/kernel/smpboot.c > +++ b/arch/x86/kernel/smpboot.c > @@ -518,27 +518,69 @@ static int avg_remote_numa_distance(void) > { > int i, j; > int distance, nr_remote, total_distance; > + int max_pkgs = topology_max_packages(); > + int cpu, pkg, pkg_avg_distance; > + int *pkg_total_distance = NULL, *pkg_nr_remote = NULL; > > if (sched_avg_remote_distance > 0) > return sched_avg_remote_distance; > > + sched_avg_remote_distance = REMOTE_DISTANCE; > + > nr_remote = 0; > total_distance = 0; > + > + pkg_total_distance = kcalloc(max_pkgs, sizeof(int), GFP_KERNEL); > + if (!pkg_total_distance) > + goto cleanup; > + > + pkg_nr_remote = kcalloc(max_pkgs, sizeof(int), GFP_KERNEL); > + if (!pkg_nr_remote) > + goto cleanup; > + > for_each_node_state(i, N_CPU) { > for_each_node_state(j, N_CPU) { > distance = node_distance(i, j); > > - if (distance >= REMOTE_DISTANCE) { > - nr_remote++; > - total_distance += distance; > - } > + if (distance < REMOTE_DISTANCE) > + continue; > + > + nr_remote++; > + total_distance += distance; > + > + cpu = cpumask_first(cpumask_of_node(j)); > + if (cpu >= nr_cpu_ids) > + continue; > + > + pkg = topology_physical_package_id(cpu); > + pkg_total_distance[pkg] += distance; > + pkg_nr_remote[pkg]++; > } > } > - if (nr_remote) > - sched_avg_remote_distance = total_distance / nr_remote; > - else > - sched_avg_remote_distance = REMOTE_DISTANCE; > > + if (!nr_remote) > + goto cleanup; > + > + sched_avg_remote_distance = total_distance / nr_remote; > + > + /* > + * Single average remote distance won't be appropriate if different > + * packages have different distances to remote packages. > + */ > + for (i = 0; i < max_pkgs; i++) { > + if (!pkg_nr_remote[i]) > + continue; > + > + pkg_avg_distance = pkg_total_distance[i] / pkg_nr_remote[i]; > + > + pr_debug("sched: Avg. distance to remote package %d: %d\n", i, pkg_avg_distance); > + > + if (pkg_avg_distance != sched_avg_remote_distance) > + WARN_ONCE(1, "sched: Avg. distances to remote packages are different\n"); > + } > +cleanup: > + kfree(pkg_nr_remote); > + kfree(pkg_total_distance); > return sched_avg_remote_distance; > } > > @@ -564,18 +606,7 @@ int arch_sched_node_distance(int from, int to) > * in the remote package in the same sched group. > * Simplify NUMA domains and avoid extra NUMA levels including > * different remote NUMA nodes and local nodes. > - * > - * GNR and CWF don't expect systems with more than 2 packages > - * and more than 2 hops between packages. Single average remote > - * distance won't be appropriate if there are more than 2 > - * packages as average distance to different remote packages > - * could be different. > */ > - WARN_ONCE(topology_max_packages() > 2, > - "sched: Expect only up to 2 packages for GNR or CWF, " > - "but saw %d packages when building sched domains.", > - topology_max_packages()); > - > d = avg_remote_numa_distance(); > } > return d; > -- > 2.52.0 > Just a friendly ping. Thanks, Kyle Meyer