From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-002e3701.pphosted.com (mx0b-002e3701.pphosted.com [148.163.143.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D68E139B491 for ; Wed, 4 Feb 2026 23:25:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.143.35 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770247506; cv=none; b=iFvaUbgsqkKy28JiieWl9zvRIAMWmN2nIc/Ez7Elm2YVXIi7HqPUBTH+T46zJMOKYrgOO05YWacvXSKES4Md9TP1euREoV00u7nbvZ3ZxXaKOsPvh6JYkOwx0Wkh9zILxJngQk1M8wfJ8IGtL0lUOgGJbqyKrN+rhKrOhK33e3k= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770247506; c=relaxed/simple; bh=CuDLtQDVsb7xgsdZgi1wvXN+PFlgs0ovMlQLfxz0gEU=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=NF2f728Yt0MIrE4Hu+ZkRZoS456OhAg46I//34fR5YQMTXK9BdlXVy0ioIE4JkbzCb/JNCCc1wTOhFp/4kPlCiYIOhrRyAo6hJ49MotvBK+9qJ0gYa9ehq/Y72Zz2wPGfDrN/VL/QMZ3KV1p2xNOvIQAO29fvDXv4/oOfKuuyik= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=hpe.com; spf=pass smtp.mailfrom=hpe.com; dkim=pass (2048-bit key) header.d=hpe.com header.i=@hpe.com header.b=o0Iapzex; arc=none smtp.client-ip=148.163.143.35 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=hpe.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=hpe.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hpe.com header.i=@hpe.com header.b="o0Iapzex" Received: from pps.filterd (m0150244.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 614N4v6d3738329; Wed, 4 Feb 2026 23:07:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=cc :content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to; s=pps0720; bh=JX0IUzk/qfY3QO3rWvpVBNCzuh OmfWNmuG/ac59HgFk=; b=o0IapzexPO6mNxZwMc8+EN3BkByj0H26Kro6zVhRKn 8ZLtQmdkFladbfmuzSPxSsmN/rcaDGHSsiqb7iPfWa1eOk+R8PFdxYcOBxOi3isa Oszh5pBjawgZrECChH1FBvOXV4QhT949hY4XJbqOFjuf+98EBHYMl7oXum1OeO8p a9CZv5jerj66y0fs9ZsdtfVLyJESkBFR2u4/0AHfskXDc1pDfFChG76eMB894qS2 O7lCAnoPN0CO7LUyCfQpXbPp4FsCBCYlLoIyRk4T2SLAXimFMJfqqszWbgQEevAu uofIC9X6hvLdnlZNM6jcaZOKUZ/bnmPlHLTS6IA8ivtA== Received: from p1lg14880.it.hpe.com (p1lg14880.it.hpe.com [16.230.97.201]) by mx0b-002e3701.pphosted.com (PPS) with ESMTPS id 4c49brc761-1 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 04 Feb 2026 23:07:20 +0000 (GMT) Received: from p1lg14886.dc01.its.hpecorp.net (unknown [10.119.18.237]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14880.it.hpe.com (Postfix) with ESMTPS id DD5FA800EA0; Wed, 4 Feb 2026 23:07:10 +0000 (UTC) Received: from HPE-5CG20646DK.localdomain (unknown [16.231.227.39]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by p1lg14886.dc01.its.hpecorp.net (Postfix) with ESMTPS id DCCC880F079; Wed, 4 Feb 2026 23:07:07 +0000 (UTC) Date: Wed, 4 Feb 2026 17:07:06 -0600 From: Kyle Meyer To: Tim Chen Cc: bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, peterz@infradead.org, tglx@kernel.org, vinicius.gomes@intel.com, brgerst@gmail.com, hpa@zytor.com, kprateek.nayak@amd.com, linux-kernel@vger.kernel.org, patryk.wlazlyn@linux.intel.com, rafael.j.wysocki@intel.com, russ.anderson@hpe.com, x86@kernel.org, yu.c.chen@intel.com, zhao1.liu@intel.com Subject: Re: [PATCH] sched/topology: Check average distances to remote packages Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwMjA0MDE3NiBTYWx0ZWRfXwCwAUcAhPnYo MBaeX6iJgEcFvL03SbK/HAlKi97LEtzf7HWj2Npj9/8lV2YEQ07IZamKgL/BboAakLG+rGP9HHZ 0yeMYIuDMprUtPjKB5XxYOOFrNFLbQhyg9j6X26ohhin9Evicl4NasZjkjBqlVKYfiekUDFNbCT +fFBIpO0OE79RijYczDQk8vqrmkPkF26C2C7P6jS1CCgKfB2DKTEzRUODevu1IZvXDG3nHoFOWN kkjoKOMhvfmOMp4p4EqU9qz2ZP10ie97bBsL+OGlNq1YN4ibU+U31IiCtk1nOuBtpqVwfxR6U6x IG9zFtE7B7shpTntM0VneoCbe7n/42XZtiwYSU20O+vV5ysOg64sBhyuGMkveuyvOPnqPvJQeE9 Y1c5X95ECncgW/qBDmsGRucQuUlf4xo5QceukMOAfrC6Qn1tzj9iYQFyNttVA92u27CVcaxV3vy FRTnJLNTtR+3Uu86H8w== X-Proofpoint-ORIG-GUID: wy3uIn4k1k23tGq9kjrBeyeTZLEi2B0B X-Proofpoint-GUID: wy3uIn4k1k23tGq9kjrBeyeTZLEi2B0B X-Authority-Analysis: v=2.4 cv=A+th/qWG c=1 sm=1 tr=0 ts=6983d128 cx=c_pps a=A+SOMQ4XYIH4HgQ50p3F5Q==:117 a=A+SOMQ4XYIH4HgQ50p3F5Q==:17 a=kj9zAlcOel0A:10 a=HzLeVaNsDn8A:10 a=VkNPw1HP01LnGYTKEx00:22 a=MvuuwTCpAAAA:8 a=LjSkC8e5mBfIJB2_I_MA:9 a=CjuIK1q_8ugA:10 X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1121,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-02-04_08,2026-02-04_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 spamscore=0 bulkscore=0 clxscore=1015 lowpriorityscore=0 malwarescore=0 priorityscore=1501 impostorscore=0 adultscore=0 phishscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2601150000 definitions=main-2602040176 On Wed, Feb 04, 2026 at 02:58:46PM -0800, Tim Chen wrote: > On Tue, 2026-01-27 at 11:02 -0600, Kyle Meyer wrote: > > Granite Rapids (GNR) and Clearwater Forest (CWF) average distances to > > remote packages to fix scheduler domains, see [1] for more information. > > > > A warning and backtrace are printed when sub-NUMA clustering (SNC) is > > enabled and there are more than 2 packages because the average distances > > to remote packages could be different, skewing the single average remote > > distance. > > > > This is unnecessary when the average distances to remote packages are > > the same. > > > > Support single average remote distance on systems with more than 2 > > packages, preventing unnecessary warnings and backtraces by checking if > > average distances to remote packages are the same. > > > > [1] commit 4d6dd05d07d0 ("sched/topology: Fix sched domain build error for GNR, CWF in SNC-3 mode"). > > > > Signed-off-by: Kyle Meyer > > --- > > > > The warning and backtrace were noticed on a 16 socket GNR system with SNC-2 enabled. > > > > --- > > arch/x86/kernel/smpboot.c | 70 ++++++++++++++++++++++++++++----------- > > 1 file changed, 51 insertions(+), 19 deletions(-) > > > > diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c > > index 5cd6950ab672..4467716f4054 100644 > > --- a/arch/x86/kernel/smpboot.c > > +++ b/arch/x86/kernel/smpboot.c > > @@ -518,27 +518,70 @@ static int avg_remote_numa_distance(void) > > { > > int i, j; > > int distance, nr_remote, total_distance; > > + int max_pkgs = topology_max_packages(); > > + int cpu, pkg, pkg_avg_distance; > > + int *pkg_total_distance; > > + int *pkg_nr_remote; > > Minor nit, pkg_nr_remote needs to be initialized as NULL. > Otherwise if allocation of pkg_total_distance failed and we > go to cleanup, we can call kfree(pkg_nr_remote) with a non > null uninitialized pkg_nr_remote pointer. Maybe do > > int *pkg_total_distance=NULL, *pkg_nr_remote=NULL; > > in declaration. Thanks for the review! I'll send a v2. > > if (sched_avg_remote_distance > 0) > > return sched_avg_remote_distance; > > > > + sched_avg_remote_distance = REMOTE_DISTANCE; > > + > > nr_remote = 0; > > total_distance = 0; > > + > > + pkg_total_distance = kcalloc(max_pkgs, sizeof(int), GFP_KERNEL); > > + if (!pkg_total_distance) > > + goto cleanup; > > + > > + pkg_nr_remote = kcalloc(max_pkgs, sizeof(int), GFP_KERNEL); > > + if (!pkg_nr_remote) > > + goto cleanup; > > + > > for_each_node_state(i, N_CPU) { > > for_each_node_state(j, N_CPU) { > > distance = node_distance(i, j); > > > > - if (distance >= REMOTE_DISTANCE) { > > - nr_remote++; > > - total_distance += distance; > > - } > > + if (distance < REMOTE_DISTANCE) > > + continue; > > + > > + nr_remote++; > > + total_distance += distance; > > + > > + cpu = cpumask_first(cpumask_of_node(j)); > > + if (cpu >= nr_cpu_ids) > > + continue; > > + > > + pkg = topology_physical_package_id(cpu); > > + pkg_total_distance[pkg] += distance; > > + pkg_nr_remote[pkg]++; > > } > > } > > - if (nr_remote) > > - sched_avg_remote_distance = total_distance / nr_remote; > > - else > > - sched_avg_remote_distance = REMOTE_DISTANCE; > > > > + if (!nr_remote) > > + goto cleanup; > > + > > + sched_avg_remote_distance = total_distance / nr_remote; > > + > > + /* > > + * Single average remote distance won't be appropriate if different > > + * packages have different distances to remote packages. > > + */ > > + for (i = 0; i < max_pkgs; i++) { > > + if (!pkg_nr_remote[i]) > > + continue; > > + > > + pkg_avg_distance = pkg_total_distance[i] / pkg_nr_remote[i]; > > + > > + pr_debug("sched: Avg. distance to remote package %d: %d\n", i, pkg_avg_distance); > > + > > + if (pkg_avg_distance != sched_avg_remote_distance) > > + WARN_ONCE(1, "sched: Avg. distances to remote packages are different\n"); > > + } > > +cleanup: > > + kfree(pkg_nr_remote); > > + kfree(pkg_total_distance); > > return sched_avg_remote_distance; > > } > > > > @@ -564,18 +607,7 @@ int arch_sched_node_distance(int from, int to) > > * in the remote package in the same sched group. > > * Simplify NUMA domains and avoid extra NUMA levels including > > * different remote NUMA nodes and local nodes. > > - * > > - * GNR and CWF don't expect systems with more than 2 packages > > - * and more than 2 hops between packages. Single average remote > > - * distance won't be appropriate if there are more than 2 > > - * packages as average distance to different remote packages > > - * could be different. > > */ > > - WARN_ONCE(topology_max_packages() > 2, > > - "sched: Expect only up to 2 packages for GNR or CWF, " > > - "but saw %d packages when building sched domains.", > > - topology_max_packages()); > > - > > d = avg_remote_numa_distance(); > > } > > return d;