From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012019.outbound.protection.outlook.com [40.107.209.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 194CE359A70 for ; Wed, 25 Mar 2026 09:32:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.209.19 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774431173; cv=fail; b=XdXjFQQ886r1Kf/hM/OlRkdU6hxKu2jgWxRJBr3q4I2mvkMq0ik5KBU0Lkf9lEc9f87lemJ7TJgdvNy2RSnUohRkpbI7/cg3gkqtwYaLP2lUeoXRBzDnnMZ4aeWxLBGuc7RgmX5cjlui47DEjyNDKiuoyEo7+CddX1iwi8+0S14= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774431173; c=relaxed/simple; bh=z+mo9pa6OcB/aPoEVD95Z1IgDlLlZ5gGB/IUkm0wxyE=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=a5NPok0fshXg/EbaAvhQqs3hrITalvPmQtOPtQ4uYVx1iNCBcIPQLyo1p5keqgh+HCByB2WxgR3C4GDQ/eiMKbD+9bogkKUv2NVH83qEEaJVJ9ykPmEXTskzyJo05Y4bkUQXt2mPhi37mesSbFmGM9YjNQZnd8TEo/EZ1+8xLnE= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=UnmxfYJ1; arc=fail smtp.client-ip=40.107.209.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="UnmxfYJ1" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=B936TkZoD/pZ2dbt6D9Ap4o6T0Y7uksogZvc/2VWBfGJN7PFSY3qCkmM12dEMS0eSzyU98xOcyX37Y4WBqFDBovM+IS1FrmRcjo8jMcmIkhxeL0TXVIo0BBmByTlh9aCV4J6B9Tf1ljtCU0H/f1HiOMNYERB45Aq5rPdSDTBMNYFxmzHXrSuFhwNsf9t09SQUXG6S99EbGkL8eMlqye1bFJ8cXKHKaDy0REF9oPlaeMyGHXzDwIukmKVWzq6O8776iyg6nObyJKEQkhS8aVc75vPXT3r4i29oZGUop7hLZWwwcJFgptWxIXqAp3UEXRZl6YenGmDmjtDuTzyhBcrdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yAwa2I6NsbYXcmwSj2vh5X11XYUqq5Yh2LOWFmOgpXc=; b=HUmlIRcCBytb626iD6c5/OgsyDbTF2WugKYuWswYVbJc1ef9OZ0/1wAI5ljt7DmjJTXRPnqJI96JjLuKNHlKDeVOGBTc4JuEL3flqNVo+sVF0xDNThF8X3BFrX5QN+ozozfl6B93VUvVL27lfaxxGDLyv9y49Jy0oxk9KgRqrM1g63bysyMc8uuYNl30GczmZiJQXOiVOylf1hvzDsp1K7RwCgBrTTRjFlFVuSQIga9D/5dixjcH3W2EU+8Wz3o3Ea4Slj60b40g/0hgnpBH+Xy1DtbS4Zl1YpmHr5+s3W47y2gnFDWlzgWLmkE7VX05Gik4pXuqZsjohwDWFOrpOA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yAwa2I6NsbYXcmwSj2vh5X11XYUqq5Yh2LOWFmOgpXc=; b=UnmxfYJ1Y6wbOQKRZd1U9VVoQxvWSkOsf8vMnhk4yXUs/7jXUkt1xNiWhLPPDPAL8YLZr982AfrSdH3XFjWU2+FpP0lHNGvKdu/nC3kBMDn5pu8uk7I2eJWF1/LrNTsvRTm5sjec7adSVvfISrp3e5CeYHXveTLIZfggyintCFn5/Z0vT5Sonot4l3qGgA3ETclC8mRvZd4uL481DitJMoRZlMkyuDsvcKAlxh8nbivF/F0VmgTDlLFm6T8N9Qs9cdf4JaihXr8odWGj42LOorLukMZjDt8W4vWVi2+cQY3ldCRA8/zeVEke7xYMsePMiY7h+4ao6lNqENqFvjz8AQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by SJ2PR12MB7824.namprd12.prod.outlook.com (2603:10b6:a03:4c4::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9745.20; Wed, 25 Mar 2026 09:32:45 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9745.019; Wed, 25 Mar 2026 09:32:45 +0000 Date: Wed, 25 Mar 2026 10:32:28 +0100 From: Andrea Righi To: Dietmar Eggemann Cc: Christian Loehle , Vincent Guittot , Ingo Molnar , Peter Zijlstra , Juri Lelli , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , linux-kernel@vger.kernel.org, Felix Abecassis Subject: Re: [PATCH] sched/topology: Avoid spurious asymmetry from CPU capacity noise Message-ID: References: <20260324005509.1134981-1-arighi@nvidia.com> <0fb05951-1f2f-474f-9f7c-9f0f15a5f675@arm.com> <86cb3979-02cd-4171-80fd-df20cb3430cb@arm.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <86cb3979-02cd-4171-80fd-df20cb3430cb@arm.com> X-ClientProxiedBy: ZR0P278CA0171.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:45::9) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|SJ2PR12MB7824:EE_ X-MS-Office365-Filtering-Correlation-Id: bf98adac-926d-4ae6-fd08-08de8a517822 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|7416014|376014|56012099003|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: AyjHkI81chWshl75WG/58v4L5/um3bUTLlJ8g6pjarclSwuI/2KLibMbSMkiznit/hfYeqKmV0dFwFG2BTpRddcpdTd52ojLF1JUEnveXGWFsMtdyt6HuJkIuaRZQhF2trfsNKgRbTf733U9zqnRSvMjK2s0NB3z/ImntryCzFVJvADg/gP0ZrODwYyMcSz/Jb0iXYeSx1WVFypGqrf4k23ws0/5PVcfMthc0xVm4RWrf8WIFbCsq8If5OvUSuziWu8SXiASv3QXQJA++HnHViZO4r28Z3PXMHU2VcnN+FexpS/pecPhuDWp2QUF3HySDlmHlU9yeqs3DNPj8ikic45ENBa8FiGzzwXVh+gbZTA5lVwtF2As5d+Z5FSCquShY7TFtTh6AVbqV3aVoiJ+K7hSn6fxGPQeH292ra2XC2UC8h3UgNo4JSjn+ZRvu6Bdvj3dQk2B88uWKnti+esTSKIMuKZ0mW9VigYMtcskvNtzn79QQ+uGsaPshjIxwk9OnYZ2RP6lPBImTWSPkbJjOA2bVmTYrt1B36WEjsiLwfxfzw7ycSwWwawUperbBPAOysAhWOJ26UrXIp3qAlPyloY5APsjFZkfTTM7S726ho16Hs08jGvssPgLoIpti6oy5sHMzEH0vcuzhNMBaSLllgF2PY0HlPe9qIlBhnkTB45olJ/ta4Y0gRLA6zSxxdZj29Wxv/zWsEbhhBoCNUPIkIlNhAh0hIjExsniATkjdj8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(7416014)(376014)(56012099003)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TDFuZTBhd1BOYWhNZ1k4bWJweS91ekluLzA1TG9pRTM5bXA3aXBEUHcvbnJh?= =?utf-8?B?ajB6aTNNTldBeFptbGJhZmNuK1dMdVRrMm1SOFRPN1ZhMU53cXM1bUlRZ2l2?= =?utf-8?B?QUxvYnFjdnFsRk1rUkhMazNkM0hadkh3TzhhdldNZ1B4Q2twS3Y1OUhsTmlI?= =?utf-8?B?REhxVitvMTcyRkRpSlZHVHpOU0pMTnhkMzhDV0cvNkhMTUs0anRKR05uNzk2?= =?utf-8?B?ME0zMnZIV29UbmNId1BQSHF5NDZUNjg3aFMzcEQvd3BiZ25NcFROS2M2MkJP?= =?utf-8?B?bVZpeGdmQ3RKWWd4MUNHcVFjWENiTWpyMDJMeGJNQjBqc3ZQNHdYdVliRmFW?= =?utf-8?B?TS9QVzRSNWRNL2Vza0NwMGE4NjEwWWM0M0JodHBtM2FhUEZ6RUIwblNDWmNE?= =?utf-8?B?ZG5VamJXYm1lKzcra2hpdFpjcmZ4dllMZ0I5Rzd3K2ZYV1RNZkZmRStFd2ZC?= =?utf-8?B?c2pOc2plcW9JUVV1RUxyVVZiTS9hQnZzQnlWWVdUNGtQM1dJYVQ2aHZML0h4?= =?utf-8?B?TE9YNEVMZmRmN3NsV0RaQncrL2pZTjlQcUVmU0U4anQ3UWFvYlR4TjAyME9q?= =?utf-8?B?QUljR3Evck9HTUZOTi9CMDZiL0pOR0Z3d0d5c0dCY2FNc1dMUXBRWC83TTJa?= =?utf-8?B?N3hvdCtvOG9WRTVWQVI4Q1lMRWJtTVZPSjBQRVowNkx0YmplbkhLNzQyTlRx?= =?utf-8?B?R3ZIc2J5VUZIK2xBWm9DWUFrVG1oTHZkN3YrWkliam1jd1hNTXg2eENJU0kw?= =?utf-8?B?UEt0YXhXQUZUK0lSSGFIdXdMeVp2dERZMys5bHBWcmhObU1NREZpQXhXNlFn?= =?utf-8?B?VkZFMWxRb3d2NFZDejVpWVJZVk9pa3pKM3hzUlpzZE1zclBaQUdpMXVZbUxv?= =?utf-8?B?Nk43NmM3ZVFTT0dqUzk1Z2JuNW9QS3pPZEdiR1NzclBqTHFITjVGamg0ZjEv?= =?utf-8?B?SmNDeWpMWHo2QWo5N3h3SFpuTkIzelJXQ2x2LzFZaWxQN3d0QnQveER2cDZK?= =?utf-8?B?TzNsOFJLam43cWNaaHBJWVp3bFlVRGIrcktkZG5iTjRXc0xSL0VqMW5sMlo0?= =?utf-8?B?VjU0RlcxVHRDVDA0Qm9lTENOQm5kTGtBUjE2UGdRd3VpZWlIcHV4U2RRQll1?= =?utf-8?B?eGJTanlqMktSZ0hXaHR1bnplNWRSbE9JeGkrZzB0TmJSVVlNcmhPSzZYc0Ra?= =?utf-8?B?SitPRmVoZGdrQzhoM1kyTFoxYytIdm5xV2JRWVNkRnpzSWJkOS9xZUdTUlJD?= =?utf-8?B?QTFBa0xsQzhiUHFCcjJOcEZQcG9tU2ZGbFJMZmkvRDlmMExjeHpqclRPbVg1?= =?utf-8?B?aWdSTlpnUjdvK0RqOVFqM3Q5Mitnb250c3VWOTF0VTE3WUp2M0dON3c4YlpH?= =?utf-8?B?ZXpLK2hjK1QwbU83Q3hMMmc2T283Y29YaEV0NWRVd1FZL0RaaE9Tam1jSXFJ?= =?utf-8?B?REsycUtMNThCb1JRcGVXS2FnTzBrM1I3ajdlaE05YThURWRiSjBmMHVEUlVq?= =?utf-8?B?RmFFeWUzRUs4ZU5zUTN1RFluUEZYZWJvaHQ0K2lxWU9zWUw5RXI3UTMrZ254?= =?utf-8?B?dExRd3FDaUFwU0h0MXNEZmFBSnc5bFkwa0NyQm1NRU5DWDJpQ3gxcWxlaEpG?= =?utf-8?B?VXhXbFo5eHVURzVYSDJuMlV5TC8zdkxkTE9NaHJ2eVY3TXhpZXFPK2JLaTdw?= =?utf-8?B?aDcvbkovM3dyU3pYVXMyb3ExTHdYc3J6b21Ob3hVTGhxRTduWk9FbkhXNllN?= =?utf-8?B?MGN2QXB4TTYxUGZMQ25WRHQ1R0xyVDcwc1I4MWc0bVRhZGMzc3lma2hHazlx?= =?utf-8?B?U0MyWFNJZkYzd3FWRVBpc25EYkZ6aWlxcmY3Z2t3cjd5SThZYXY4aGxiQTlK?= =?utf-8?B?M0pHSys2ajkrS1hDalF2WVRFL0p4emc0c2U4KzhpNStRdGwzY2J2cVhlNlRG?= =?utf-8?B?YVVSaXdkRVlDQkRkNk9HaXdYc2RrVG1uMDNoZWZCK0FPUUdpNWh4YjVpTUhk?= =?utf-8?B?OHBUcFVFVlJ1czBHbGpiNUVZamZ6dGxmYTZZZU04VjB0Z0hRQ2FsdG0vSU1s?= =?utf-8?B?K0FlZWNDYWsvSXNqaWxNN3lpbWhEMnJGYmJzV0lBU2NqaEtQWGxmSzZtdThj?= =?utf-8?B?YU1PYUMwcEY4SlpFQmFTc0diaTMrMEFIZURpK3ArMjY5ZVpKMHhEZlV1MVBk?= =?utf-8?B?QlJpNDJnQzNSMEhSZ2tQZ1gwMnhJSXI5QnNtNTQ2ZFE0Y3FQTVYvYysxUS9s?= =?utf-8?B?MzAyWVViclZOOWxBYnBYY1dncGNuaXlVbW9keVdvTkkvSmZyT3doZFN0cnYx?= =?utf-8?B?N3kyMVVzZ0MvRnJQL2pUa2h5VEJyODZKaGc3YWdDanZQaUNEN0RlZz09?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: bf98adac-926d-4ae6-fd08-08de8a517822 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2026 09:32:44.9259 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PZ132Z0joyCzeGlOM5is7SysFxZ2rusE1BMe6XMGFtBrBGgvmLrXWuQNV9eXtpCW6TcLQ9IdFS5rhxgKLNhLXA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7824 On Wed, Mar 25, 2026 at 10:23:09AM +0100, Dietmar Eggemann wrote: > On 24.03.26 12:01, Andrea Righi wrote: > > Hi Dietmar, > > > > On Tue, Mar 24, 2026 at 11:29:24AM +0100, Dietmar Eggemann wrote: > >> On 24.03.26 10:46, Andrea Righi wrote: > >>> Hi Christian, > >>> > >>> On Tue, Mar 24, 2026 at 08:08:22AM +0000, Christian Loehle wrote: > >>>> On 3/24/26 07:55, Christian Loehle wrote: > >>>>> On 3/24/26 07:39, Vincent Guittot wrote: > >>>>>> On Tue, 24 Mar 2026 at 01:55, Andrea Righi wrote: > > [...] > > >> The first time we observed this on NVIDIA Grace, we wondered whether > >> there might be functionality outside the task scheduler that makes use > >> of these slightly heterogeneous CPU capacity values from CPPC—and > >> whether the dependency on task scheduling was simply an overlooked > >> phenomenon. > >> > >> And then there was DCPerf Mediawiki on 72 CPUs system always scoring > >> better with sched_asym_cpucap_active() = TRUE (mentioned already by > >> Chris L. in: > >> https://lore.kernel.org/r/15ffdeb3-a0f3-4b88-92c0-17ffb03b0574@arm.com > > > > Yeah, I think Chris' asym-packing approach might be the safest thing to do. > > > > At the same time it would be nice to improve asym-capacity to introduce > > some concept of SMT awareness, that was my original attempt with > > https://lore.kernel.org/all/20260318092214.130908-1-arighi@nvidia.com, > > since we may see similar asym-capacity benefits on Vera (that has SMT, > > unlike Grace). What do you think? > > We never found a good way to specify a CPU capacity in the SMT case (EAS > and energy model included). So comparing CPU capacity w/ utilization, CPU > overutilization detection etc. definitions get more blurry. Hm... so should we just avoid calling select_idle_capacity() when SMT is enabled to prevent waking up tasks on both SMT siblings when there are fully-idle SMT cores? > > But in case you now want to hide these small CPU capacity differences from > asym-cpucap setup you won't run into this 'SD_SHARE_CPUCAPACITY + > SD_ASYM_CPUCAPACITY'. > > You still will have small differences in sched group capacities but this > is covered by load-balance. > > BTW, you should have seen on Vera ?: > > sd_int() [kernel/sched/.topology.c] > > 1720 WARN_ONCE((sd->flags & (SD_SHARE_CPUCAPACITY | SD_ASYM_CPUCAPACITY)) == > 1721 (SD_SHARE_CPUCAPACITY | SD_ASYM_CPUCAPACITY), > 1722 "CPU capacity asymmetry not supported on SMT\n"); Yep, I've seen that. :) Thanks, -Andrea