From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2056.outbound.protection.outlook.com [40.107.101.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70E552E401 for ; Sat, 25 Jan 2025 14:40:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.101.56 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737816058; cv=fail; b=tHYhg7gPVHX4rW6hG/Di7SmOHrpl48rjPMVyBGSE3ICzfahK9WOeqc0hx2QcyWDH89wktV7Kf/0rjDcBI+V0SP49EgfraqJdnpvN5Ykt3YccDZ3OghhrXqHYTOMSTVIkSYcu0USs5Q36Pbfk5CWqC5oKv9b/dDT/HZdXOtqHQjE= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737816058; c=relaxed/simple; bh=ZkiIUCoYQBJ06V4vyxjoGs76/oPx299WvQib4blp4m4=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=a3/joPmR4IT8igfEqEoPdJ11TmPjdWQouanuufz6GCTsht9QK+vT2p+9FvoouLOYgQ2f322iGfAgVTxwIyViRk3nTCGU9Y5MOCEQ6SVSzQe2JqIOugBay4P1rc3RpY0JtJh4dLNmtz7Fobe5RP3tZJfSRxllzesWXnDzEWnrIso= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=kh6Ig4jF; arc=fail smtp.client-ip=40.107.101.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="kh6Ig4jF" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kU998r6H21IWs12i1REVcywNikL4zIutgHozRMF6FVOw4FzaU8NK9pZEFobdwuLaFIsxoS4NxKas0ONeS87473jcE3JRndiN2MrKFKoJXr1PHuFxY7NQVcuZ8RaUiglJaVS42LKXa2i4DDxmzJkkHCtOs0INjeeu0UBxfTU/M2J+/fd7EI+xA7OPQg2skzYLaKrrEeTh9s/HNYUAlUo+X2C7JYxFxj1+MaMXzI6qYQegIxF04hktlA0RWgIWpsDPPLIniALaQtgYUYole3PoZ6a148UVzu+ShdGoKfj3XaHDFoy5ZqyXXJ0YEyXTn5/CQakoKb1wbE9X/Zi+5plSEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7R0lrUWj0XtS4YtAGgbiQHfpJQ/UFa438+Gzd8ykJsc=; b=pDj5XwSets1M2ZujykFo1IJQy+ufVwIAaqNGzF/zntoNGm8SYbJCn4ZyuuieIYW2XxHxQHxgCqiY7um8hiCgbqlQk30DKmp1m/Jmg39MZFp+SmKf1zO8vOLFzjm79ULyuRBKel4EVv3L7eahWQI0hxhRbMHdoiRX7rF1wtvTp0p03HBgKZ22pTprW5vfFEy6sYbotadV/CcgC9IoYalFDecpgAWLpvHSnv76xPIOfa6iUFrjQqpfqCjjjHbdJ/9xB9k4q5gAN8t+KxKvcCXcY2yMWH2YDN0kYW32yjUYK+/j/YWHJtwH9/6QqNhbsrejUZ8K3jYUNthdEMIcq9cVPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7R0lrUWj0XtS4YtAGgbiQHfpJQ/UFa438+Gzd8ykJsc=; b=kh6Ig4jFHCPhbaThQjcIJk7c3lkslQffV8qKMnu2HfgcD9xGYmGSbB+QPfD0zeh3Nn6Y9GpcDG3RahKf8E4r1W2wAPJ28dZ7GMMc2uJkAaRIBaIEGYlpoYd45ijYfqp06miPK5KsjqOd1lLNkZeOJdBGR6PCN69QPKa94o3gskzOKgS+BvAVvvoJWMLHFrHhcPNnEzCA00MwgeYgfsSbq0FSniRX9TdhIgyb6cAAcDVm2yb6kS0vGdSIVM6fen1frq2/dhdrRytDj7EWf2O1Dpogp/iwbVUAw/m4M/FXrKXLBm6wnnfP5Vx6Wr2FKM7QNouI+r7s2WkMDuI+mpywPQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SJ2PR12MB8979.namprd12.prod.outlook.com (2603:10b6:a03:548::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.20; Sat, 25 Jan 2025 14:40:53 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%4]) with mapi id 15.20.8377.009; Sat, 25 Jan 2025 14:40:52 +0000 Date: Sat, 25 Jan 2025 15:40:48 +0100 From: Andrea Righi To: Tejun Heo Cc: David Vernet , Changwoo Min , Yury Norov , linux-kernel@vger.kernel.org Subject: Re: [PATCH sched_ext/for-6.14] sched_ext: Move built-in idle CPU selection policy to a separate file Message-ID: References: <20250123220202.16274-1-arighi@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: FR3P281CA0157.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a2::13) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SJ2PR12MB8979:EE_ X-MS-Office365-Filtering-Correlation-Id: 6d169579-8fa9-4ab7-ad89-08dd3d4e44d0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?k0vvIJ9uTEy/bpFe8usblsBwoXvSkVtOWaB+1q4M28tFHTVlmnZ9QlMPNIiH?= =?us-ascii?Q?kMcKkYYoljldmic6s7vdGgM09KQmNKUNsHQXYMO7xy2W/cjuU73KXnIICJl0?= =?us-ascii?Q?Dg9yyly/RG8eZ6f7UMSUsxOlVJVzTvDzJhsGsq6GS+HsXEhKC1TpKuzuZeiJ?= =?us-ascii?Q?cfbH2xdFPaPQsGTH056o+CJuWANrtO2iRiDGE9Sg0g9z2SGFYai66sF8psfw?= =?us-ascii?Q?mtOxB6nSanpW4jcR0/MV3rthzoPsIvSFzLRhQpxy/ec1APt0xTMvr4bN9pjv?= =?us-ascii?Q?gZF32BcAY1O1ucZxKq2AiBHpayiaxzz6jWzvPFVehtkW/EDNI0EJXXwnCEtm?= =?us-ascii?Q?nFnVRWPpWxyTqnpeviu+iyFBBgCv8ycO0hN0SSe8k8a+HwAbpzlFQyN5nIBk?= =?us-ascii?Q?/PSVrZSiaXDku9yz6FbGw4ZGgZtSt3NjCumCeMOzQHzniClmyDBX54xgZ7aQ?= =?us-ascii?Q?vQSEXXmnAwdGeyrUC6omCeQJG1BHoqbTeyc55yqA6135nNVxXPQKX4vwDO7F?= =?us-ascii?Q?qDxTCOJS3OpKn1gP9ChNHy+45Hz7RBW5iMWLYNJs708cJlpSgr24dh27BMOq?= =?us-ascii?Q?YU/eydq7HXUvAiDw9C5qjG0pnuo3zjBiNI3Z7kkLwajI0xgwE0jSvspWR7PE?= =?us-ascii?Q?MqSBvsXHWOHG8PswRZQ2IK9XTR/e86URSPOJ4pZoU4F5kFAfP4JFS64z6LR0?= =?us-ascii?Q?DyZijjP/yBN4ndtpVauTDhoeMMzGoN9VZwNnaHha5Motf9f37LIghrRWShfu?= =?us-ascii?Q?Wt5jfB+atFIsKjk/cgR+uhRMsnXfkv6ZU1p9QEXJw3rAxHpptgKH9Ah5EYv1?= =?us-ascii?Q?2Kd9EwnJXIgn+Z5zurjBDDU7DdAibNtcxSgME//FlByS9DL+l6XZNavpKF+e?= =?us-ascii?Q?OeneZgzBl3B+1pjV76ebPftypnitQPOEdgc6z+fiZ0JtmUSZjHJwtWWAwJ7b?= =?us-ascii?Q?2nQItZc6ikk39nVjITqFcvWQFbSpr3kD5B3DD4gVBsmE6ph7dr4IpIppGGyo?= =?us-ascii?Q?4e7s6ixnXe4+ZYGTd1Amss31Q6SByAVO+v9PR7gL8gkNxIVbEdjHHkZcjezf?= =?us-ascii?Q?o6VfFo9uqp30jtr7rmJAjQxCTzSt7Hs3c/IhV9JLm18u+Dtn777PrFnLhjAL?= =?us-ascii?Q?0IhLaH43OX83sz4i+fQy7KkrFCJQLwt30BT+VeLtpwrWNEP4JyI60dl5KwZv?= =?us-ascii?Q?ztCQhZfbEAOIztuCXIh03exQbwhbpvFtUg8jmlrm3wMEv2gj2bvDApRgYCCl?= =?us-ascii?Q?5ejOto+nsOAqY05BUqZF5rqIy7+/eNz0XrCP9DIsRIkgtba89uxgjEvwqGg5?= =?us-ascii?Q?ahpF/iRvxvminEojoG91VSJZfTumiKY2JFSCcLrdwdX5V6m4nuWh/4Hd+mnQ?= =?us-ascii?Q?L0lWaY5B/ompd4x9+A9CZdRSF/dG?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nqPkfIUAhn3RQ5vhsw0HO24y8BBysSHo5PkZcnrImsdrDt57IgJBOsSoeIZN?= =?us-ascii?Q?QNCs14yyRkaXn9Kx/DQsmrtpwE3VylCFZ5uI5/Z7jsPDACX7HcoCsXEe9Hmr?= =?us-ascii?Q?GA8vUzWNOjzBchsCR+WTybpFpZGpa6UuPDQTT5DmJz4KpWBzFSeVa4fKuCYh?= =?us-ascii?Q?6gSNjAqkDiJH93RaKsZG0QKJk7YWqqVqRcL9+E8rhK+kd3As5tMRvspgTgwj?= =?us-ascii?Q?oUh6f4sGhk6twekEpWKvod6Y+iteDeQ6P9FtuRnH2V/6BOnOl0ps8EX1HkEZ?= =?us-ascii?Q?HIhvva5kbsnynNqLMIi2eCZdVS/iR9nvRYbsoOX5KUUfksQFK0cZTLGCblQv?= =?us-ascii?Q?t92lw4z8TN/DjqkBXtt4uLqPYiLjWv4pDle0BXvszNSb+XMlqMpynznD/o8m?= =?us-ascii?Q?S8OQRIwHrImoY9gNmJZ61iLSJMKDx6siEG2Pm8Y1jlOVjPT5RW2pImms3Qcg?= =?us-ascii?Q?6iy3mUcQyeVN8kNDt9TRIjvpKAV/wkHAa0bGvQpXmR4IqYelhqNZIYbH498a?= =?us-ascii?Q?213bnkKAtEGcT61Ex1XixBgo3aRqT2xrkJDqj6QsdM/3wHy7VwA6yoRJ1fg8?= =?us-ascii?Q?lELYDCcspJCXi/JuqklWG1tk435z7q2Ix+EU5fxd7dLEcWqSHgYJt1qF+NIO?= =?us-ascii?Q?y9Fy/Iawj/NaLZgPyDPcpuBp8B95pbayorTPtjsMWVDGXW0Nx86NhQbk/l6g?= =?us-ascii?Q?8mnm3PHrSmlE6Jo/GsmRT0fTU38uSnXn/f8LauEEXbpylIfgKi2H6SyFuB6h?= =?us-ascii?Q?5VTn+ySlGobVwhGywA/++s/2jAURRpoornaAjKvt7kISiM9+Qy76aiWGiXoO?= =?us-ascii?Q?/BTaokWUWll6FT5eW9xftgSAYQjeB2S76uRGHej92a/AtZBLpQ4HMdWbkMN7?= =?us-ascii?Q?j34LU0pUUE0y2tgPP4kq2nzroqQd1f14MRd5uBW0ehGFXET41d6aSIFuwz+k?= =?us-ascii?Q?40D9fiaymuI0gNygcSPYcoKuoIVcVSDXhfHiHM209vFcX9jCzLJit6siA6Fa?= =?us-ascii?Q?Momyhprfx8UC9k8Dek/LcrF803+xUcjEBoqiRv5k7Xe4X5SKkvnhd7uZJDQY?= =?us-ascii?Q?35gKAW0woEJJhytgSBDpnL71sL3yihQzywvxbQE+iKHwyDto1dyzG6GCAlqP?= =?us-ascii?Q?EuSSSbpKXrdGkIU+kbUkOYotXUYNrMLAw3Zz9J9GHZfSDh9fHdtDwU+EmfHM?= =?us-ascii?Q?13kZZtMKoO2paFRj6V7SJ5tCNA5HCzr/1bM3pQPqsrr/YhigcBYjSIGrHSY0?= =?us-ascii?Q?efEPl4CYXcIfMT892P30AHe11+TpUunLiwE5jZg7l8mPeF92xFfdnyEWTcN0?= =?us-ascii?Q?KNuO3WGOUGK8jtOkEAbFZioQm/diTQziPOE9kROSKeg23xCn+1dbp+Mn5EFy?= =?us-ascii?Q?4UVqtQ2g44D9ZOIr/hFDG8aR5PhMH1E1hTdu1EHQ6x9D22JodU90Zm8H3QoH?= =?us-ascii?Q?xFg7hjLIXDNV+6/YALJyE9QbVOhg2RJGNWPG6yQtMrTXIHEsc5igonFZdFpG?= =?us-ascii?Q?olBvRy+tgQYLHFzSPoHaK8zdxauXKDOW5e/sjb46Tfs6HfRh+2ycmebBoO/W?= =?us-ascii?Q?F7ACm78pfZ9E+/PWKhho2QyQTOSXWcpZVA+2ZKl4?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6d169579-8fa9-4ab7-ad89-08dd3d4e44d0 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2025 14:40:52.9081 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gqHQXQ2POC+lYHN/I6/eBqXcGiAd9M1LjmFrbr0fibroMV4tlNK1wDXlMmallUQvNgExpDJQ6h2TF4Vjh16LfA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8979 On Fri, Jan 24, 2025 at 09:51:18AM -1000, Tejun Heo wrote: > Hello, > > On Thu, Jan 23, 2025 at 11:02:02PM +0100, Andrea Righi wrote: > ... > > diff --git a/MAINTAINERS b/MAINTAINERS > > index 023df277737d..cd3d5b139a11 100644 > > --- a/MAINTAINERS > > +++ b/MAINTAINERS > > @@ -20935,6 +20935,8 @@ T: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git > > F: include/linux/sched/ext.h > > F: kernel/sched/ext.h > > F: kernel/sched/ext.c > > +F: kernel/sched/ext_idle.c > > +F: kernel/sched/ext_idle.h > > Maybe kernel/sched/ext*? Ok. > > > diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c > > index 564f250e7689..a24d48cebfb7 100644 > > --- a/kernel/sched/ext.c > > +++ b/kernel/sched/ext.c > ... > > @@ -896,6 +893,17 @@ static struct static_key_false scx_has_op[SCX_OPI_END] = > > static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE); > > static struct scx_exit_info *scx_exit_info; > > > > +#define scx_ops_error_kind(err, fmt, args...) \ > > + scx_ops_exit_kind((err), 0, fmt, ##args) > > + > > +#define scx_ops_exit(code, fmt, args...) \ > > + scx_ops_exit_kind(SCX_EXIT_UNREG_KERN, (code), fmt, ##args) > > + > > +#define scx_ops_error(fmt, args...) \ > > + scx_ops_error_kind(SCX_EXIT_ERROR, fmt, ##args) > > + > > +#define SCX_HAS_OP(op) static_branch_likely(&scx_has_op[SCX_OP_IDX(op)]) > > + > > This chunk is no longer necessary, right? Correct, no need to move this chunk. > > ... > > @@ -7750,12 +7037,6 @@ BTF_ID_FLAGS(func, scx_bpf_nr_cpu_ids) > > BTF_ID_FLAGS(func, scx_bpf_get_possible_cpumask, KF_ACQUIRE) > > BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE) > > BTF_ID_FLAGS(func, scx_bpf_put_cpumask, KF_RELEASE) > > -BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE) > > -BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE) > > -BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE) > > -BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle) > > -BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU) > > -BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU) > > So, these were in ids_any and could be called from any BPF progs. > > > diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c > > new file mode 100644 > > index 000000000000..ca99fc58af91 > > --- /dev/null > > +++ b/kernel/sched/ext_idle.c > ... > > +BTF_KFUNCS_START(scx_kfunc_ids_select_cpu) > > +BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_RCU) > > +BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE) > > +BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE) > > +BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE) > > +BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle) > > +BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU) > > +BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU) > > +BTF_KFUNCS_END(scx_kfunc_ids_select_cpu) > > But they get moved into ids_select_cpu and can only be called from > struct_ops. Also, we currently don't use kfunc id sets directly to decide > which kfunc may be called from which ops but that may change in the future > too. Just create a separate ids and register directly from an init function? Oh I was missing this aspect. Yeah, moving them to a separate block and registering it separately is probably a cleaner approach and we can have all the idle-related kfuncs grouped together (without breaking anything). Will do this. > > ... > > diff --git a/kernel/sched/ext_idle.h b/kernel/sched/ext_idle.h > > new file mode 100644 > > index 000000000000..c1385af1ceeb > > --- /dev/null > > +++ b/kernel/sched/ext_idle.h > ... > > +#ifdef CONFIG_SMP > > +static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); > > +static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); > > + > > +static void update_selcpu_topology(void); > > +static void reset_idle_masks(void); > > +static void init_idle_masks(void); > > +static bool test_and_clear_cpu_idle(int cpu); > > + > > +static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags); > > +static s32 scx_select_cpu_dfl(struct task_struct *p, > > + s32 prev_cpu, u64 wake_flags, bool *found); > > The way scheudler code is built is weird but let's keep it as close to usual > h/c splits - make them extern prototypes and prefix them with scx_. Also, > maybe name them to explicitly show that they deal with bulitin idle > handling? Ok, I'll apply all these changes and send a new patch. Thanks for looking at this! -Andrea