From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=pass header.oldest-pass=0 smtp.remote-ip=40.107.209.60 ARC-Seal: i=2; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=pass; t=1793986456; b=sbxuynyKM39zq1GOwx8oQzrWvyqdVTNxszubpO+UiBYmGvTZYUcOG+IVPHS656En31TI RG/3y/K2FJwjMnbOVdrdP/HZQXD2Stl+6lBP2z0HfO16eJItcYiHOLehtyWin1rE/XSB2 89oyQxHwuFO2yuIUo0nbpxXWkY/ZR6QVbPhJAnWNGRyFexVDm3w+3S7XaIrPOPdigwxPW lGL4p11xBLk+p3H6d0u/koKQpGrjlEmJG96flHJmBuSgqWwy9O7nfDcmuwleLYFeL7cQc 2RxoN7/4jJaIS7Zj6GqJEBPjDfbxiTIGmUhF6O+5yaB+u4tpvme8hecaAyKYnvpIJDA== ARC-Message-Signature: i=2; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793986456; h=DMARC-Filter:DKIM-Signature:Received:Received:Date:From:To:Cc: Subject:Message-ID:References:Content-Type:Content-Disposition: In-Reply-To:X-ClientProxiedBy:MIME-Version:X-MS-PublicTrafficType: X-MS-TrafficTypeDiagnostic:X-MS-Office365-Filtering-Correlation-Id: X-MS-Exchange-SenderADCheck:X-MS-Exchange-AntiSpam-Relay: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report: X-MS-Exchange-AntiSpam-MessageData-ChunkCount: X-MS-Exchange-AntiSpam-MessageData-0:X-OriginatorOrg: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource:X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped; bh=zr5EsqAeRgl0PR9xLhThn3KngTsDBbfz9y5uLfr1pNs=; b=EIjgTckAeBCaOvpxhsO/+dd3SLTjXLeDbUMTvRhHzwONOl/0VZyqsYUnhyo1FLbRijgD ttOwvLjk3Dvp8jzQecfoFWxIDbI0QjbOKmk6B7mHiUS56RP3s0aHq67niVE+jdHHLiHDn GOIXzXbzfFz3soxmj7tobo+BGVNOl4EKOoXiXDEs5bt2k4lqMa2pK/Zay8+sPdN6+YNY2 4KLD11h2Ughj1VUGns9a3BEz35cywgmer53Kl7dvpBq+1lkQCVj4Taty6aNhJUSx1tiW9 tFV7ma1++Nx1WlWcsKok5QLFX8HmXrGHd7zjLYw7q/MWSzrAGC16PFINj5k3c9LsM/A== ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=XwCHydSH; arc=pass header.oldest-pass=0 smtp.remote-ip=40.107.209.60 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012060.outbound.protection.outlook.com [40.107.209.60]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (secp384r1) server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EDAF9606A32F for ; Fri, 6 Nov 2026 17:34:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org EDAF9606A32F Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=XwCHydSH ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ycKSAJeI5GGMKL3yd0cgX04/QYJddRke7vGmbgNkOSDoYmozTHWhldvu/egiZRIdmJmFAZSNUE7O4ElSkbnfkOjHs5MdKMYj5dj+FLqmycb4/IQeQxwpFp3ULes2xUrd4iLFnARQETcFP3jcaz9DqD9nIVTn3ebvTX+avWUI9WCVeyjSgzyvgomhfACqO8Bxq3Dj7aC5htJ5wM/M7aT3qrd0a1Lb7mq/ilqXLp7IJPCYlfGue2xXN/xFyi2W6hhgZoTmC190DSW3fLdXDmCME87UfzAvo1IEGSNXjrBMAJyMQ8Iz6kPpeoYn+OlY+LdMX1ohe7cBLB/kwwJQgLudCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zr5EsqAeRgl0PR9xLhThn3KngTsDBbfz9y5uLfr1pNs=; b=jWsnaiMoX4I2VWnxw5z8fWFhKUVWGhONz8W0iEcjcnqrz8RFfZSqiyqyfYOp+O1x3We/LbHtJkxaNHovcUw+Arh1+Hj7NQacN0uIy1RxL6OZh+NFL+LWuqN39H686OIFy0EuMQP3YGHqUI2WLetP69cCimBI7CFEJustrUgeAfjUbFi5e7g3J7kG+qtbWg5Icr/86OCEKLirEJnVjVmlvdAyOCVgRwqNnNRcAhghgQhkZ1aBEhdiuo0T8kzPRiIqNe2v37RJZRbtfjn0t8JS04eiaSh46Iaw9RO0fz4etvJfWq33TJ+14y/AMI52a3clJ3UNYNFSYAQqS/u6OMaDtQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zr5EsqAeRgl0PR9xLhThn3KngTsDBbfz9y5uLfr1pNs=; b=XwCHydSHGe0jOlEoJ2oXmByZC2WOWAeI2qjpxmBLRZvHmJnlczajMZR0yjeRoW7xIM2AbXBaEaqAl7eJTd1TysVcuA4kVEH8Hj40aw9xWIgq6jxCu5hR2EJSq6lvZ5gVNbSns1y3RaE3JCgMCBk1sYzprxd8FpRhrUHwDyEMW5M3YGJa0ZK01s8+yWuThosYImjKRFSLNmmxBcAuvp8tJnrvQLz+Nvj6V2Zafk/Da4a+9uKtKP4v6k0N5y6ZVg3bQy85f4YZnJxCcGZnhCOCFsqotwCtJlVOhsPAsNopj/mtKANeepR2n1JEq34dvNpnSKYuszHJ8f7x4y07frIfEQ== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV3PR12MB9356.namprd12.prod.outlook.com (2603:10b6:408:20c::21) by AT6PR12MB717548.namprd12.prod.outlook.com (2603:10b6:170:10f::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.11; Fri, 6 Nov 2026 17:34:11 +0000 Received: from LV3PR12MB9356.namprd12.prod.outlook.com ([fe80::1c36:31b4:c420:6286]) by LV3PR12MB9356.namprd12.prod.outlook.com ([fe80::1c36:31b4:c420:6286%4]) with mapi id 15.21.0592.010; Fri, 6 Nov 2026 17:34:10 +0000 Date: Fri, 6 Nov 2026 12:34:07 -0500 From: Yury Norov To: Peter Zijlstra Cc: Ingo Molnar , Thomas Gleixner , Juri Lelli , Vincent Guittot , Shrikanth Hegde , Yury Norov , "Rafael J. Wysocki" , Alexander Gordeev , Andrea Righi , Ben Segall , Christian Borntraeger , Danilo Krummrich , Dietmar Eggemann , Greg Kroah-Hartman , Heiko Carstens , Johan Hovold , Jonathan Corbet , K Prateek Nayak , Mel Gorman , Mete Durlu , Randy Dunlap , Rasmus Villemoes , Shuah Khan , Steven Rostedt , Sven Schnelle , Valentin Schneider , Vasily Gorbik , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, driver-core@lists.linux.dev, linux-s390@vger.kernel.org Subject: Re: [PATCH 0/6] sched: Expose writable interface for preferred CPUs mask Message-ID: References: <20261101192203.6979-1-ynorov@nvidia.com> <20261102084704.GQ1779021@noisy.programming.kicks-ass.net> <20261105145537.GP1813469@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20261105145537.GP1813469@noisy.programming.kicks-ass.net> X-ClientProxiedBy: BN9PR03CA0296.namprd03.prod.outlook.com (2603:10b6:408:f5::31) To LV3PR12MB9356.namprd12.prod.outlook.com (2603:10b6:408:20c::21) X-Mailing-List: driver-core@lists.linux.dev List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV3PR12MB9356:EE_|AT6PR12MB717548:EE_ X-MS-Office365-Filtering-Correlation-Id: 8577c57b-a7b3-496e-7021-08df3c2c2ade X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|23010399003|376014|10070799003|366016|22082099003|18002099003|6133799003|10067099003|5023799004|56012099006|4143699003|11063799006|261026121527099003|261026140227099003|261102123227099003|261028144727099003|261022105027099003|261022115727099003|261021135927099003|261021121427099003|261009223027099003|261022075727099003; X-Microsoft-Antispam-Message-Info: Bk6udFt41qfC9pkfEJd/qLNkQcpM7qRlAqsbAGheitOsEbIgsFGrh+24CoaJsOB+gMkvuFrC7YPX/RmVcyc/aYNsSA4nypAvrTY/pclT/kOtLufNoYGruaLnfSihNXHyBqQEi4KHnpqrEWhMX3SASHR5HNatCci2cJBMoOpNx9aoF9gaRHo8V9HYJadYx7BNDNRuBzSOXFAr1SGZ1kIjvte6oeiYMZl2Vr1nBQpW6RhRSktACz/fmHNPJXeoG9IH+eDI8nj8/DasWYSzW29M9JzUYldERsYqDbgf0o7auMWnDxDVfKBBjAvyBLJeVnahiClQPpNVweryCH/kBcePFZRffCD4Prv9FPemstx2bMtHylNAUVamsoegVBFhrJkpAi4FQoHJv//UI/BDbw2vHnibigi53UN5+zql2lulXVp/ZS3TpGXmrQvzdUiXU+BIv8qjgqqXlrCX48OWiGfpttK0HifkIgbIiVtn869f+WNirCbIm3mdMpaQXkrDzJWIDG38qwJDEeYRpQT3YPXGdkMr3SFguHy+9LYanb6UrLOr8g6npCtdDKaCF+mZ8HQ2fat+7AauklVXwb6+H48vW5SI9uB4OHrQz6cJ/4w8wON5mIRHKadYSgCTY0lF3CH/GGQR7Bp5hrWRh7plEI74/NVKrkwCFzZBdwQKYjhr8MA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV3PR12MB9356.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(23010399003)(376014)(10070799003)(366016)(22082099003)(18002099003)(6133799003)(10067099003)(5023799004)(56012099006)(4143699003)(11063799006)(261026121527099003)(261026140227099003)(261102123227099003)(261028144727099003)(261022105027099003)(261022115727099003)(261021135927099003)(261021121427099003)(261009223027099003)(261022075727099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ONj6MYRNU5GeOK5xNGOR8syM0nxEh3ZQ7Vmv/z79sFfpwc5fD7w2mAEIrvOS?= =?us-ascii?Q?y1VsgfHdu1MhYz2sFvBd9ScybniN20gu2+c/lvkQKNBZ2O7UrwZXqLc4DmQG?= =?us-ascii?Q?pzjcR6ePV4PA6lUbUf0ioYPvlQc3vusfNZIxQind4BLV6X9sysSVoywseAWb?= =?us-ascii?Q?5/rovejGWgknz2PFH+Id9nXsPBc5aZCbpk/Y1czY9PwMENRvu4T6wWYTUe4m?= =?us-ascii?Q?g7DzPXVWfhuuHP6DgKXJIR476pbkCJS+Qpm03/GyVVhMv5WmjwqSS8WEWO7u?= =?us-ascii?Q?2k8Wn63m9FfpXSMFLbBahCbSU35d7gueQ/wXtQv/ZTENn/r72K64NoZzHYXs?= =?us-ascii?Q?z9jIcaotVXSkvqW1gggzK7H/Fa9XDYbxhxNCBGC95cS4eIC7B6lj+Gi/CWUJ?= =?us-ascii?Q?Tx6RU31lkS+Fu5cEvhEhwhyPjyjaif8l3/MH+PrKECQMfV9LbrAMmCHNr5D2?= =?us-ascii?Q?Zw/Q++fkhWLy7WrVCtXG+/bttsudxzt5vrKmySfiJpeixBjcETOrnUHjrqKi?= =?us-ascii?Q?kUrP1gE1Ws6vEzH+k5g6xXr29pQhAsOb8X13FkGMXxBVT41ov1ZIHxcU/9Zo?= =?us-ascii?Q?wyME82/RY/Y6qFr/en3VViBUcU1eBMjJ18J6Cdrc6OASR0hICXA5xZTRCJBp?= =?us-ascii?Q?bek/7zSXBlgFNRheVMur37ny3DW81FxlaZSQ8DbrvSDCjN8X24IQftpbJehv?= =?us-ascii?Q?3FwxLNuyLHn7T4ROicD0PCVBHTLnvwnnS1/VOKFmw2+XWXUwdCDzsCjhxXUi?= =?us-ascii?Q?fzn+xURPr7ZL3AsWUjrqBpeCw/S0PxWFIzVehxZO3i2AHH6rJuN5vh1GrZxn?= =?us-ascii?Q?RTR/aIaczAji0daVYgyMfFw0n5/wOlrJ20f0iQuzhv98IaZ0TgvQ+kCnOS6V?= =?us-ascii?Q?QKz4W3YrIbMlG4/xeEv7nsJxKlMJ/JWb3rnGtZJkqWAYBn9+aQvqQPSpBc0C?= =?us-ascii?Q?Q+U99D/ctbKd77jCDcBu1W8gHQZA+zfClELG2Eox8r9dGk0H44dxabylIRXK?= =?us-ascii?Q?fG6x0zYZdQ3YyiAPh617hbeapKEmdWrqxzcdyUXlgWkY4BFQC+Uuuk3Hnxx9?= =?us-ascii?Q?IJrrZ9RJKh/YdkRS4X3fYkfBMuJYPvLlpuppiJ9APVmZvFDOaTplnWgdvyEc?= =?us-ascii?Q?lABghcdU9WSI7o3yonGHlTNpS4L0tBCG/gHIvJFeorQ3vXlGcOkvsBIbEgdP?= =?us-ascii?Q?kYmjcWY5UUCyai8h7y66VkOa1+TF12X7SK1q6ZriVo6fBwc69iVa2i/xqfw6?= =?us-ascii?Q?+1PEdSKqfKo8VY4yKmU6nLkFDs17M730N3jOt0enjBtyu8LWPXD4/git2MsB?= =?us-ascii?Q?zAUOEeyKitk1paTqQOawvuyreLZeKqPt4stzleZz6cJC2KF1XJ3o3alnpyhJ?= =?us-ascii?Q?fzXb2hETafge+gC9rtp7zeKo+IAIKr9fmWPb9A1fPzq7ylxTWqMq8Xmupx0m?= =?us-ascii?Q?HGC6Q2TNiLyBr4CLnkUmn6V5V7cRLmNIlwEUrbja+MZ5ED0S90KHembQuDuP?= =?us-ascii?Q?B05C/626b5IRIMLxf9SFW0Hb02MfmdFlBx+ISnHRtGtuKaaB0q4WZO/4Bmul?= =?us-ascii?Q?pjujw4ln5qK1+uIp6dikWRT/PiNfSwc50mysu+Dc5qZxHwC8EMswxVWuIkEw?= =?us-ascii?Q?mGfwB2VPoPKeL0MtnAXPz/DMbWwTUMD+BN4iguypTMXWjuMJSC00NAQJJzly?= =?us-ascii?Q?p3qppgJutPvELnuuvpiEcrh8h0g9kOogA9tBKmq9uYihPyJkLu+73+Ov6O1i?= =?us-ascii?Q?0u3Y19PARg2noG8yjuAYhaJNxYiPO/SmwyOJHln8tUcbkjfIQv54?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8577c57b-a7b3-496e-7021-08df3c2c2ade X-MS-Exchange-CrossTenant-AuthSource: LV3PR12MB9356.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Nov 2026 17:34:10.7989 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /k2FgHNyjlUeWvVm7jF+9eOK69UBPvKyUktiF6XS/mC7vvVJTpFTLWOHL5KvDvxFETA1rSLWDuNyX/dPm9riuA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AT6PR12MB717548 On Thu, Nov 05, 2026 at 03:55:37PM +0100, Peter Zijlstra wrote: > On Tue, Nov 03, 2026 at 11:49:02AM -0500, Yury Norov wrote: > > On Mon, Nov 02, 2026 at 09:47:04AM +0100, Peter Zijlstra wrote: > > > On Sun, Nov 01, 2026 at 02:21:56PM -0500, Yury Norov wrote: > > > > The preferred CPU mask currently lets the steal governor reduce vCPU > > > > demand when physical CPUs are contended. This series adds manual control, > > > > opening more use cases. > > > > > > Sceptical, I am. > > > > > > > For example, setting CPUs aside for selected fair tasks at runtime while > > > > steering other fair tasks toward the remaining CPUs. > > > > > > That's exactly the kind of nonsense we don't need and makes me want to > > > stop reading right here. > > > > OK, I started with the wrong argument. Please consider this as the argument > > #3. > > > > #1 is: conceptually, preferred cpumask improves performance because it > > moves the scheduling decision from host to the guest level. The guest > > can do it better because it knows more about the workload. But then we'd > > consider moving steal governor logic to the guest userspace, because it > > knows the nature of user workload even more. > > I am confused; it is a simple control loop and has nothing to do with > guest workload. > > The host is over-committed, this results in vcpu steal time. The guests > observe steal time and reduce vcpu utilization. > > It is a co-operative scheme; a bad actor can game the system. It is > imperative all guests behave the same or do not play at all. > > If there is no level of trust (our coconut friends) do not use this. That is correct. I'm just trying to let trusted userspace to be able to make a final decision, not to cheat the policy. I believe that the preferred cpumask approach may find more use cases than just a blind vCPU throttling. Even in case of throttling, the userspace might know more about the nature of external pressure, workload and VM priorities, etc., so be able to make a better decision on the preferred vCPU candidates, beyond the steal time accounting. I can imagine a host running two VMs - one for performance critical workload like AI training, and another for latency critical inference. In case of vCPU overcommitment, symmetrical throttling of both VMs may substantially affect user experience. The better solution would be allowing both VMs to communicate to each other and decide how many vCPUs should become non-preferred on each side. Steal governor is a reasonable default approach to the vCPU overcommit problem. But I also want to be able to adjust it to my local needs, if I have to. > > #2: convenience and scope. Some systems don't allow loading modules - > > either on config level, through selinux, or somehow else. They don't > > even have compiler installed sometimes. > > You hurt if you shoot your foot off; not my problem. If you run guests > like that you don't use this. I don't configure systems that way - I only request shared access to the particular hardware, i.e. to run some tests, often for just a few hours. Sometimes those machines don't allow loading modules, sometimes it takes few iterations to reach to admins and ask them to update drivers... > > > we already have cpusets that is more than capable of doing this. > > > > Then why do we need the steal governor at all? We can steer all > > processes away from a given vCPU without any driver, only by playing > > with each individual process' affinity. For vCPU throttling, task > > isolation, or whatever purpose. > > Different level of operation; different semantics. Cpusets is for strong > bounds on your own workloads, this preferred mask is weak bounds on the > system due to outside pressure. > > In a way it isn't much different from ACPI idle injection. The idle-injection indeed has userspace interface. Powerclamp exposes its actuator through /sys/class/thermal/cooling_deviceXX, and documents userspace control loop. By the way, preferred CPUs approach may prepend hotplug in the overheat defense chain, as a cheap but weaker alternative. Thanks, Yury