From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2049.outbound.protection.outlook.com [40.107.220.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35CE91487F4 for ; Sat, 4 Jan 2025 07:06:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.49 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735974405; cv=fail; b=NeCUIkeUiaW+UqCdx/Yj/ophnVEu54fvuOotgLtrBuJoUllvRGF1qdc4aByOF9iE7KGzBD2Gh5BQuP1lrSdsHXGTsavnfzXsP6UpnAo7sSL3YevdACDk2Us5VC0hWCRN6Oc5BtdGeIef8aAR14MFa9XsEaBFUB3CGOCIpwqKCfU= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735974405; c=relaxed/simple; bh=nBL/7eifJ22DM2R25AOc27zda8sDpCQpg/e8H3wZOGo=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=MtwnSGG5mXnlF2WsTCYGm4D3Wm6J5DdwcY3EfMuviZM2XHiQg5osUuvZUMRVV0fkvWFiEjCQvdTWGnSRaRz0cqLVA5XdRJslOvJ7s+IpO0UPr/BNK6DyYfbXBNO0eBAIFGa19BhbMoKxPUTOIS48bB0A5INaVm0GdLgbWjTkjjs= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=swQgMCcw; arc=fail smtp.client-ip=40.107.220.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="swQgMCcw" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qbN0/rQl58UhLAq0h262icCgWl7SNCDfQ5061srPXPN0l1DuD54FnlMHCdK5DR6+wNCrfIdeWNOX1yckiVMIaAGJYSFaMAAyqpMpt40a2tiXp/XTp/f+g6ukDdxde0VjqoQ64o0N9c0BLTmPEftPhClxl4exL9qTErvR13JutXkk5hY5EmQGbili1yxseElidsvRsWupjSa5JATvjZFF6AuQKvB1XyVsL4CKfxgYBmF6ptoZwwzYAqkzsOsCydeDi9HWXv9oxcsQV8evnSoasHJ5dtgcjBjwCeLb3anTHTqMKKyRiUu+93iAHnWoFGO71OIVp13TDZVyAJayIEijFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MRZP2uiNgyebgjafONBEYmc2TlXXdzwY1K7+nb4BR2s=; b=Ugqu0ynbb6ZtRY4k14Li6PGBKa6AuqmICYXx4RgrSkf4XL3asjLqNswKEQKwTNWEPORK6B8IxRn8o0Vnyra6HBzLTM4qqtwaVy/naUC4IN0FD7LMJiGo3o6O+nz2J7mxNUv2uwsxD7rrQ5ec9577JM3x+KInLgtJjKqKu/KjLDUcdpufwJsxY4EirgE978zcsD478i+Cyh4fdnSmP3Hx9j2Ns1tZTDpVuMyCBNd0lthV50PHWzCKbhQMEy1uKph/Gw8Amtq+CqDBTM/yRCTLf/XRVXqPnoyChNbu2nHwWwPrTYNQfhTr5YJzb4z+B8JmaG5tj9E9c78DzlWOJqhwPg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MRZP2uiNgyebgjafONBEYmc2TlXXdzwY1K7+nb4BR2s=; b=swQgMCcwxdjST/cxG+1CY+HEL/YOo7KCmIlaYZSRm5ReN8VtA/a/U8Ass5DUZqDEjTC/Fsp542VHQbiq6q7pG07esZzhLR/l10LdMKtCRyHXuBY7r5o/ygWGAQ1bMZifbd2NYRlBr62iHpFHrOzyNkd37camQGaUe/nJWX5r+ceLeenI5MoXbLFFR70cKGLNyubLAfRhmaWOWTiHDhagT6ZoXF91uR+XNqNBtcbUKc4qGtIODQkbE8+qjsJxGJm919ecHCnMw0F+m8TdkqIh9Ubh6dwsA7kc7F+BKIEnSmJAJ/fai6dy/beCEuccuNXPuUHDlUPmC9voGKoULtKkIQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by DM6PR12MB4401.namprd12.prod.outlook.com (2603:10b6:5:2a9::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.15; Sat, 4 Jan 2025 07:06:35 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8314.013; Sat, 4 Jan 2025 07:06:35 +0000 Date: Sat, 4 Jan 2025 08:06:26 +0100 From: Andrea Righi To: Tejun Heo Cc: David Vernet , Changwoo Min , linux-kernel@vger.kernel.org Subject: Re: [PATCH] sched_ext: Refresh idle state when kicking CPUs Message-ID: References: <20250101182449.21517-1-arighi@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: MI0P293CA0005.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:44::19) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|DM6PR12MB4401:EE_ X-MS-Office365-Filtering-Correlation-Id: c3d8aad2-27e9-44c9-1f98-08dd2c8e538c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?W+P2hBED6Qbj/ycokVqz2ZfrYaIDf5fIWbhl4TBTlVFbwGKSdV99/o79+mzz?= =?us-ascii?Q?66YqRmraq9wzHxeHLzMWr9oF10OiTnR+9BD96x3bERTzE8Eei0owPiX9M9Mj?= =?us-ascii?Q?f24AUeCpAf9wYXVQhfa6o9Tw7Ex2qBlQuJKx4LBfk1wEnQrF2Fu4mBy6ez/D?= =?us-ascii?Q?H+GH+jstgbhIcJQOdYUOpRc/j+P3sCpUdk3xsrWaCp+2IGKzFcWlIyKpuwV5?= =?us-ascii?Q?0X3oOElwkLlRzDmiirkvrJ9ZA8MozvxwidW2IOVvD6cGvo/iFfBuabIyABWs?= =?us-ascii?Q?2s6Uz9P7iSqEMneE1+XZ1RKuDvsJypawABgX7FQ+EwertEFEhxc9mtFGt7bP?= =?us-ascii?Q?OFd1Aq47ZFDkWPP1+jUluJpL5bfHbMjdwcJT2nfFSshQoxlbhvg2ZU/YYtgA?= =?us-ascii?Q?A4xTriB3mvrrqWc1uzVGr0giVGxxT5N8r4pHAUblhnxlKiHLr9ZwETEv8UFE?= =?us-ascii?Q?sdUzo0a9YfljnJ89NzVIWqojrtgfxev/ljC9UrPICF2wdOHeuN5VSiDOUyxn?= =?us-ascii?Q?CEcK5bOlL8n2o4Ise9gmE6U+mYZoCQpOyiYoEkRVeeMShYFd/2sbXDvw45rg?= =?us-ascii?Q?6wFGVjLeKDF1bf2DDNBVPSoFIUZlZhwAx8claSafjTWSxXbTHw67Y6x7W1TD?= =?us-ascii?Q?o/IHlaTK6p6bdaYShjHkhrA2jkssT1a03gn1PDpsZVPrLtVgkHUkTaBiXt6v?= =?us-ascii?Q?idlzgaP5OqI/Jv9KxqEuTlTMdKch1FYBOq6vAdArTBs3QiLu5h/N8L4kZ54U?= =?us-ascii?Q?teeRXP25JfFVClpV47/yfiECvkcInwfO2C6EjAp8DF3G+AFKfRMPmNmYSrjf?= =?us-ascii?Q?WtVT7Ku8bL10/SCV138Zsm/eDkKvTm4YNjmxTfkowK4fIsHFjCpUqsvAH+kp?= =?us-ascii?Q?+CbCH9sgxX3ODriwuh60szGSWzqYRN0m1UCLpuWIe5DzWDYedMTpOkxJ6GVI?= =?us-ascii?Q?DMS4dq+vzZc1g/LgwkCdR6AlNzJ96wb2i+JMcM6FLsDl3cFQAUVozUSiHdIP?= =?us-ascii?Q?eMsfTaluaaKTLmBZECpVURHkyofFoW+kpqSwZbFFkxSuI9BHj/y3bRxAQLk5?= =?us-ascii?Q?ReYotgJfBnVHhJce81iTegSbcT/RVzn+EioePFn4U0lkVa67np0SMlFvtfcD?= =?us-ascii?Q?WO1ebM/Qayx4t2XVXVShMOydDV7HkpFEQvTkcBuf4T7hCnnRpPozn0b7gzYc?= =?us-ascii?Q?hUDuM4uct/VmW4W/oG1LzhzRIoUAmDMZMfHxkmeVyXg/6Kcn9ugIzkYeZqBl?= =?us-ascii?Q?cd5GOMMbJthslFZE+VVyQJhP/zpnLg9fFw4mL5hZcq15nzRb12+v7SvQDH8a?= =?us-ascii?Q?EG+zN0BJUj8CopAmWzFu7TQfzoVh1gMvCX/MrHUgpS9U7rf6slVhub2J2O0W?= =?us-ascii?Q?lpOdVmQ/0+hQAOCc+tkUYYSCKC4O?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?aiN5LWn3CFZzPYYV5QjgEYNhoZY8vA2S3rTqoFN79PsZBfwJx4Q6K2ZHkikJ?= =?us-ascii?Q?g9onZ2XidbI/iGudycS+KE/PUgRsY4ltFPQTmPutw0vm4Xrmqq9OC1vM7JR1?= =?us-ascii?Q?e+GsnF+28V5atWfTItQZxF/aIl0GKGspYWvXh312wowjQ8aQr9T9FF4PCVai?= =?us-ascii?Q?NfTBPpsZylk5dHxX5qetSG5wPsF54w2C8oJBSZVUbWj4uTCnNXRFjNSjJx+t?= =?us-ascii?Q?EbZ1UXa+dbNC6hUmHOWRWAhGSkQXHQSEB8yq5D0HYs5fxF0x7howvBQ3WPsk?= =?us-ascii?Q?dTWkjXPUqDOtDML898xVPTJxG3P+S3/OzTPq6UiT8/68apavlJcxWYsJlwJW?= =?us-ascii?Q?CgezTuInyidBzb81y+3UsmFePuex5Yf3K2hl1teNRPkx0r6zCDmvt1QaHuY2?= =?us-ascii?Q?mOwKm419WiEZ1MVeTjcGOyvI4VbVVHyjVXQocJjEQBfCvdncOrm1+FjaZQLx?= =?us-ascii?Q?hcwjyvo4Yr1/MSzn0K6HZDo7WlcT/TqlYAm/enhGgfp8baRsQyN3oHgncDz1?= =?us-ascii?Q?JlKkCM8muH8KSi7EsFkBPs9LzITD3WEr8U2KaIXbtcYfvGo4ANGLU+uHYcs5?= =?us-ascii?Q?37+jyGeZ2jJrYl3TyIPmNbEz4WUgy9LFBP2Zn5z43KopbpzB1+RcGN88C5Qb?= =?us-ascii?Q?BJPIWkOq7X9KeyrllLelyVAsZTZSqQTYc8gDvig9NTqngeFlljOAW7m4x04c?= =?us-ascii?Q?aWpLZtub7wqLnRRn95Go+6NGHoIUcEUJ2LzCEpdyO/n+lUrWt/GqrGArQdmG?= =?us-ascii?Q?aMQNTRgjAeLFIWMqIu31/MSRMiwc8JnDK1KCprrDcJHM3GvvatSs8HTxqN3Y?= =?us-ascii?Q?7i+OZdF4Rn8vE5R565pVrPz5Lr0EK1ytqCx905MDrlAJKaRgEgGlYrcGw7+H?= =?us-ascii?Q?xK7QoRZitCuCnXzR43kBsicaS2qfkbog1oS19JJP5rw4LsxP1YwGI4wemlqn?= =?us-ascii?Q?x/xtHfJQwzpv7XSn2n/0QNMWdTDddO1lCkXhGopBYPT3ovUxyLboz0iFP7RW?= =?us-ascii?Q?6vtTd2WYHc2VhQ4x4pFAr1SVeVYmSjBMB/7pGwe2BFhpoZTU3GughNcgBeYX?= =?us-ascii?Q?Cc7UY+Is1/rWGkYKGgVmad8Uh/es3nsbk9bkb1gVoKumpV0FT7S1Np4cXZTP?= =?us-ascii?Q?zRoVlFvueo2yvvsx0c+0xNuEnfPq7YS/OF/+gxCgvByt2xCVEmUuL3TY0y+u?= =?us-ascii?Q?RTdASKPVOXp5WkT4gBtYcnF81Meq1I+Nyse+DSWDGT8zhVyZjmGkgj5SekbD?= =?us-ascii?Q?ESc5FlRnDeSkE2J4KL7kZ4wYvB09O6fD2Cgf3Qoi9NzjT+VtseP0UwTnMZw6?= =?us-ascii?Q?VvKiDdKVqVNJVg0/UC+7urZuvEq0+pC5dPD1yF3uB4pqUWJNVck1asTS9v30?= =?us-ascii?Q?zU5cwkzc1R8ZutjBX+CLjvmn8Er0LAya6lceqfBHtwUICK/TpRvWlVr+1C+G?= =?us-ascii?Q?ZCqvhFv6eSd7eO6iVCUwhrnbem9P+cBF4ANgdOTZY0Tjd7Wu4JzrMU+mMHeZ?= =?us-ascii?Q?W3PQR0/CKgZv3aGS5oRuTBCfR/B0DweHYhSOGb/rMGwmQaLsmkirmB+kMrhr?= =?us-ascii?Q?5kw3wUmLR2t4rzqjG4bUvaAp7089JwmjeSh7Voab?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c3d8aad2-27e9-44c9-1f98-08dd2c8e538c X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jan 2025 07:06:35.6803 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: b5oevics3kRtLU9Dd7vZLJD7JTio97qcWTj2XsGfJF6EFs20gc3xJEA48BBzqDYYrQ+J7ybCts6lrV7ggrW+XQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4401 On Fri, Jan 03, 2025 at 11:39:51AM -1000, Tejun Heo wrote: > Hello, > > On Fri, Jan 03, 2025 at 09:55:14AM +0100, Andrea Righi wrote: > ... > > > When the put_prev/set_next paths were reorganized, we lost the signal on the > > > CPU re-entering idle from idle. However, that signal is still available if > > > we hook into idle_class->pick_task(), right? So, if we move > > > update_idle(true) call there and make sure that we don't generate an event > > > on busy->busy transitions, we should be able to restore the previous > > > behavior? > > > > Which is basically what I did here: > > https://lore.kernel.org/lkml/20241015111539.12136-1-andrea.righi@linux.dev/ > > > > We didn't fully like this, because it'd introduce unbalanced transitions, > > as update_idle(cpu, true) can be generated multiple times. But it's > > probably fine, at the end we would just restore the original behavior and > > it'd allow to solve both the "pick_idle + kick CPU" and the "kick from > > update_idle()" scenarios. > > > > If we like this approach I can send a new patch updating the comment to > > better clarify the scenarios that we are trying to solve. What do you > > think? > > Maybe we can solve the unbalanced transitions by tracking per-cpu idle state > separately and invoking ops.update_idle() only on actual transitions? We could call scx_update_idle() from pick_task_idle() to refresh the idle cpumasks, but skip the call to ops.update_idle() to avoid unbalanced transitions. However, if a scheduler implements a custom idle tracking policy through ops.update_idle() we might face a similar issue: the typical sequence scx_bpf_pick_idle_cpu() + scx_bpf_kick_cpu() + CPU going back to idle state without dispatching a task would leave the CPU marked as busy, incorrectly. The issue is that we call ops.update_idle() when a CPU enters or exits SCHED_IDLE, whereas it should ideally be called when the CPU transitions in/out of the idle state. So perhaps a kick from idle should trigger ops.update_idle(cpu, false)? Still, I'm not sure if that would provide any benefit... after all, do you see any practical scenarios where having unbalanced transitions could be a problem? Thanks, -Andrea