From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CWXP265CU008.outbound.protection.outlook.com (mail-ukwestazon11020133.outbound.protection.outlook.com [52.101.195.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99689481650 for ; Tue, 25 Aug 2026 14:14:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.195.133 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787667261; cv=fail; b=ULT7xT4nTcNou3iIqV2LuS7O1qDCIpQussFwVAQdkov/m60wAAAE5T0BL9qpbLlnQWLDoS96kioqvw7zYO7L8jJYLCXKfCTcad6Cdq2EO7e+hcqSLlrVNrwgNHppBdyKORG3TpiRTyCv0NIKK3QzMHkEI/lvMehTeWuJQpiFLP4= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787667261; c=relaxed/simple; bh=/04n7Rklv3vsBtw9V0KImzvkzUNR/ihHFstg0xQn+ws=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=NZV765Y5KsBUuKqjn9MECtXUanViXmEfDrQVLustf3IiaoozW/TKLQQG8DWFL9Ww/tYOzR4rJaRUbVxG8SN2Vxne4XXHi2h/td5U+3bE3vEpqN82TTExhZs+5GwJ2Q58vk6g4QyWx/LHucv5IeC48Gu3NDBQM8Z3pwxF5mwAxJI= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=atomlin.com; spf=pass smtp.mailfrom=atomlin.com; arc=fail smtp.client-ip=52.101.195.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=atomlin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=atomlin.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=b+24q8x4cuNx/UXn28oBW0lt+i5f4Y6rpyRrRB2iAFY2Qfs4PkQS1wJnuMVJAKNCuKWlQm6GHgxsWlsECpfyNijffU8nyzmdUoHsu9xx532JJSlBURLjMoyNBcGheGLDckeF81Z9/EGzgKEEbq3/0RYVVnJAJGFdtn6gu8BhQQI1T3hwkdqjmp7f/YnoNjCfFwDzYY1GyXVNE7rj9luTTW13D4Ofce9gS/nNCPSUZNXCWFHAj+hQS5rAInnT8MneaEvsrAjowGj45f3ZzdrGuBl0Zd7i47wExa3tVgfRLd197awkMqI7FmPM9cgDmwXlvIqLv1yN3yds+G5T2a5pgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:MIME-Version; bh=17JQVpo/+BsVuU+LUtteUok96dPjyPiTZoer+3IVwiU=; b=s5vqFDSlg7qg6+JYnC3CJTLeNQQpX+aW7qLyY9x6VlBmqsRkwPlb3JsjLS3DdcKnvjIbz1473LErybyEwbCGCw+8H3OiTM8kM+QfaOZIlytB81WhmgQkForsejH7molmyYQ5mGMhNcAkoFuUIP5KMwquAf2KkKj/UEN77j3DUQr1cWEv2uL9qbmf2K0QqtCIIypG2GXguCvA60T0nkIp6wPsDaSPXYBIPR846+rHXnbLPj263InGly1Ff/ad5PY1ajCJKcuc7RmWHNGApjKZ+rE6WQPBWr/jGoJaC0ZVDwdu48CFewtayhipH3s5vYFg23sXadWDtxhpaQ4b6iYq+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=atomlin.com; dmarc=pass action=none header.from=atomlin.com; dkim=pass header.d=atomlin.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=atomlin.com; Received: from CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM (2603:10a6:400:183::5) by CWYP123MB8971.GBRP123.PROD.OUTLOOK.COM (2603:10a6:400:286::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.6; Tue, 25 Aug 2026 14:14:16 +0000 Received: from CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM ([fe80::cec4:77ab:262e:d230]) by CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM ([fe80::cec4:77ab:262e:d230%4]) with mapi id 15.21.0339.012; Tue, 25 Aug 2026 14:14:16 +0000 From: Aaron Tomlin To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org Cc: dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com, zhanxusheng1024@gmail.com, neelx@suse.com, atomlin@atomlin.com, chjohnst@mail.com, mproche@mail.com, sean@ashe.io, steve@abita.co, rishil1999@outlook.com, linux-kernel@vger.kernel.org Subject: [PATCH v5 1/6] sched: Annotate rq->rd with __rcu and update lockless readers Date: Tue, 25 Aug 2026 10:14:08 -0400 Message-ID: <20260825141413.868997-2-atomlin@atomlin.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260825141413.868997-1-atomlin@atomlin.com> References: <20260825141413.868997-1-atomlin@atomlin.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: LO4P265CA0200.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:318::15) To CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM (2603:10a6:400:183::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CWLP123MB6607:EE_|CWYP123MB8971:EE_ X-MS-Office365-Filtering-Correlation-Id: 1d2db888-0a6f-457c-8702-08df02b32579 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|7416014|376014|366016|1800799024|10067099003|22082099003|18002099003|56012099006; X-Microsoft-Antispam-Message-Info: 8EJyyLHYbceE3jTVCZMvTrH9BsKBqK5SuyHt5plHmtQr5nVVn50ehJfkxH6a1a9bi1yo205n0gZT6yExuK/CLhS3vucRb3Z4LncHt/1PJVWlBcYmd7pgvsK24WygW8GahfnYwba2cBIykvCaJKkEQFn4VadzH0463zdg6LopDHKoQx0HFvFrZPZ63evav9Uj+GI4JIy4t6JxEtz66z227jXLmGGWochPqhI8rJPuzXi+FgtaIZZFf8RqALkDNVxCXFB/qAILDJJJGv5lEzGjPw1KORtbYCjGgoP2qXmyhceV4GyvVGE8vq/6/09GB5ZyQDALaGd+WOqxL9jNjkvVUoKbtP+6pwFegpJPZetlewwx3GpJu+0CIzGMA9tzAIOjboY7ZeWrDAWyKZMuRC8fJBPOzoIsJWJdUeOQHYp6QeGX2Kl1mrMNFn21h/r92Odb2kMyiWYJxpCrS650JGhfxdcuUTJscC+V1HaLxkwvBLL8yjGS1NsFCcQ6UwTTp7l3aMrj+YSxMC2kC65yHdMy7Of56FXXsvYVoXuWWYUMt21Z8o7xeXaZnrznJyuZNE/YndYjC1XFFTSGRrCXo3XLs1IUPQIn5+9xrA4a6+GZBWBDW8vBjELluSZuG0kf0AW1kgSdHp3VlsPTQ8hMNIleFNSdhWr8ANxsADyuos4bC44= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(7416014)(376014)(366016)(1800799024)(10067099003)(22082099003)(18002099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?5mVHKRVp0SllOcTPikWFSp/Xr/LVPKPRlEZSeYGV7yDB67pqaQX9AAlTeObb?= =?us-ascii?Q?cXHqDqXrLTFNRg8KcyQ+Fqv1s9qOKNSzZtTBP0F5wPllbolRaDeYr5Dmb2K1?= =?us-ascii?Q?ua1Rfo1OHeXdKFRaKNYucglViTQnPQHXBmA1OCf+OBbY8TzIN9Vn1nZUax3Q?= =?us-ascii?Q?kasNLF2jxbNlKaRIpufldLLDa+LylYzJjnNWhcKNC8F3yBUj6lSMwBE2XsCy?= =?us-ascii?Q?gyZ074g919Nyei8mR+nBM2yYjiyWWYg/EaJwbP4GWGKi5mFW+zeFFjqMkbr2?= =?us-ascii?Q?Ayq5YK00U1Kb3yIe1XXr29857BWaoNahgbhuBVpsNualaHop8n6KaJX093Ue?= =?us-ascii?Q?oWZPDPFvia96tgxeEjZPZxhG34srJDWseVtg935R6hcu1T/dtRn8YYeVwM4m?= =?us-ascii?Q?D+ndEdganBh09C3B53U0TBTUFvGUAiDleAhysIylh280BphsCQWBRRO8pRVF?= =?us-ascii?Q?cFsdWZoXCHbjc4Kr2STSERE/tWRiWgF6YBHSbmFSSSXOmMcRd+XSSO251ltP?= =?us-ascii?Q?X5raVvZEs+y+fZKUbryMKsTSQIjShpvYEnkrxAd/MNzT/1KL4HZGi4D4YWw0?= =?us-ascii?Q?zCDwfFK9Lux5asOyEKnFx8N/Y21hCTV53qNOihbAVPYFE5Jxmr/D+1gsd/rD?= =?us-ascii?Q?F46e6GOm4aKwFNyLLbKdIHqPnSWwAcOz7YQ83o1xGkdWGmvuBhpqjn05beJw?= =?us-ascii?Q?KpLckA9Dt0I0RucSbcSGSuAeb6qdW8U5gHj0MOVigzT9LZ7Urvn6nH851HZi?= =?us-ascii?Q?pfiN4rSWCLW0W1rFyvKFc95fhYbSiXawZOhZOw0Sb/PNkGumMBKd77JGFqK+?= =?us-ascii?Q?aEz0e1dao6gjxpN57v0xcj3hTXNNjXgL2uhBMb/lBzFAa0/418nyqm/PgSn5?= =?us-ascii?Q?lv5MHhuJBdbEutePQeMy9lZxW1CEnvsQ0ljZW8s63x0fJrkAPcxC3lL51URZ?= =?us-ascii?Q?o67MPeJ3cbFG2E/RRSWsRy90ysgV0BT5LT2n8BA87TgJT3sx7nbbOrF1cYJG?= =?us-ascii?Q?aJ6n5JsuuObliwYHJX5gNSmCnKxY6NzjXcch/ZI5lOYHwoVWk67VkmqftOM2?= =?us-ascii?Q?fMm4mkNXsx7tMs83uFF+Zwn7cNRKFAIzbNr3ULY2Q1gjn//Rr01HiKX/2qu9?= =?us-ascii?Q?gntAAgmS2LqdpPgjRR4t/UqESNILboDWYBLiQl9qLaRJcX7lwnWP0c5O8ttV?= =?us-ascii?Q?Wv2NQfWq65Lv533Yol9mTNoilblZmMusm3OL+imezInR2TSTts+COh1AKMLv?= =?us-ascii?Q?DmrT41Mrjen1qdO1YnnXxlz1x/N5sJAINXsIxzlrPHrmHcHxz5uMPBcAW7cT?= =?us-ascii?Q?/63IwRMM/MeN9w9sa7QkfXwiwWXfCpHm7F39Sye/XZ5DWmQpLLvXHEdOErZd?= =?us-ascii?Q?+PbGAK9cP85pbNtR8nDV8WLNlc/wPeRfRD27yYxmuzr7dVHKtHvG0pXv5yJA?= =?us-ascii?Q?oD8NYFUiIMULoP4isJ/sAmBgxWUOvRYMS4lAR5Q66P2cBIa24seiOfvytdCG?= =?us-ascii?Q?bKFopRj2tID775NkR+1CpY7Qrtu/Q8aiQKWyKam0VTd4+c1cMAPSrjmqgSxO?= =?us-ascii?Q?nOZqqyft0YBYHRpBun0MFgcpU4HZ+1GilFU0HH3zcKcxRsxQviNohjhdhOSq?= =?us-ascii?Q?kfCH9jFg0nJlg6wxx8r6Kf6RZuQ6gCaoVUAHdp9vhqqpC6UoA8DW/YaqfihE?= =?us-ascii?Q?R7Q12u3cXHfLac2VoR+Pjc/8fKrJjdTdebssnVr/fT8Uz4V2?= X-OriginatorOrg: atomlin.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1d2db888-0a6f-457c-8702-08df02b32579 X-MS-Exchange-CrossTenant-AuthSource: CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2026 14:14:16.2348 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e6a32402-7d7b-4830-9a2b-76945bbbcb57 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ci64ateeuZ5QXoGOozcOaGQrWSMcRawSqUGzIbv788o0Uk+gxhGIz8W7Veih1zNNYnlmF0dJwWG7M+cmgRoRDw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CWYP123MB8971 The root_domain pointer rd field in struct rq is updated dynamically using RCU, and its memory reclamation is deferred via call_rcu() in rq_attach_root(). However, struct rq's rd field was missing the __rcu compiler annotation, and several lockless readers across the scheduler subsystem accessed rq->rd directly without using RCU dereference primitives. Add the __rcu annotation to struct rq's rd field in kernel/sched/sched.h. Update lockless readers across kernel/sched/ to use rcu_dereference(), rcu_dereference_sched() or rcu_access_pointer() appropriately. This ensures proper data-dependency barriers on all architectures, enables Sparse static analysis validation, and documents RCU read-side ownership contracts. Signed-off-by: Aaron Tomlin --- kernel/sched/core.c | 16 ++++++++++------ kernel/sched/deadline.c | 8 ++++---- kernel/sched/fair.c | 29 +++++++++++++++-------------- kernel/sched/sched.h | 2 +- 4 files changed, 30 insertions(+), 25 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 96226707c2f6..3882aa99e2f1 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8580,10 +8580,12 @@ void set_rq_offline(struct rq *rq) static inline void sched_set_rq_online(struct rq *rq, int cpu) { struct rq_flags rf; + struct root_domain *rd; rq_lock_irqsave(rq, &rf); - if (rq->rd) { - BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span)); + rd = rcu_dereference_protected(rq->rd, lockdep_is_held(&rq->__lock)); + if (rd) { + BUG_ON(!cpumask_test_cpu(cpu, rd->span)); set_rq_online(rq); } rq_unlock_irqrestore(rq, &rf); @@ -8592,10 +8594,12 @@ static inline void sched_set_rq_online(struct rq *rq, int cpu) static inline void sched_set_rq_offline(struct rq *rq, int cpu) { struct rq_flags rf; + struct root_domain *rd; rq_lock_irqsave(rq, &rf); - if (rq->rd) { - BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span)); + rd = rcu_dereference_protected(rq->rd, lockdep_is_held(&rq->__lock)); + if (rd) { + BUG_ON(!cpumask_test_cpu(cpu, rd->span)); set_rq_offline(rq); } rq_unlock_irqrestore(rq, &rf); @@ -9012,8 +9016,8 @@ void __init sched_init(void) #endif rq->next_class = &idle_sched_class; - rq->sd = NULL; - rq->rd = NULL; + RCU_INIT_POINTER(rq->sd, NULL); + RCU_INIT_POINTER(rq->rd, NULL); rq->cpu_capacity = SCHED_CAPACITY_SCALE; rq->balance_callback = &balance_push_callback; rq->active_balance = 0; diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 0f858b98c9aa..8e1e8337aba0 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -122,12 +122,12 @@ static inline struct dl_bw *dl_bw_of(int i) { RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(), "sched RCU must be held"); - return &cpu_rq(i)->rd->dl_bw; + return &rcu_dereference_sched(cpu_rq(i)->rd)->dl_bw; } static inline int dl_bw_cpus(int i) { - struct root_domain *rd = cpu_rq(i)->rd; + struct root_domain *rd = rcu_dereference_sched(cpu_rq(i)->rd); RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(), "sched RCU must be held"); @@ -159,13 +159,13 @@ static inline unsigned long dl_bw_capacity(int i) RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(), "sched RCU must be held"); - return __dl_bw_capacity(cpu_rq(i)->rd->span); + return __dl_bw_capacity(rcu_dereference_sched(cpu_rq(i)->rd)->span); } } bool dl_bw_visited(int cpu, u64 cookie) { - struct root_domain *rd = cpu_rq(cpu)->rd; + struct root_domain *rd = rcu_dereference_sched(cpu_rq(cpu)->rd); if (rd->visit_cookie == cookie) return true; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1..ad367a542eb0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7739,13 +7739,10 @@ static inline void set_rd_overutilized(struct root_domain *rd, bool flag) static inline void check_update_overutilized_status(struct rq *rq) { - /* - * overutilized field is used for load balancing decisions only - * if energy aware scheduler is being used - */ + struct root_domain *rd = rcu_dereference(rq->rd); - if (!is_rd_overutilized(rq->rd) && cpu_overutilized(rq->cpu)) - set_rd_overutilized(rq->rd, 1); + if (rd && !is_rd_overutilized(rd) && cpu_overutilized(rq->cpu)) + set_rd_overutilized(rd, 1); } /* Runqueue only has SCHED_IDLE tasks enqueued */ @@ -9358,7 +9355,7 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu) unsigned long prev_delta = ULONG_MAX, best_delta = ULONG_MAX; unsigned long p_util_min = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MIN) : 0; unsigned long p_util_max = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MAX) : 1024; - struct root_domain *rd = this_rq()->rd; + struct root_domain *rd = rcu_dereference(this_rq()->rd); int cpu, best_energy_cpu, target = -1; int prev_fits = -1, best_fits = -1; unsigned long best_actual_cap = 0; @@ -9562,7 +9559,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) cpumask_test_cpu(cpu, p->cpus_ptr)) return cpu; - if (!is_rd_overutilized(this_rq()->rd)) { + if (!is_rd_overutilized(rcu_dereference(this_rq()->rd))) { new_cpu = find_energy_efficient_cpu(p, prev_cpu); if (new_cpu >= 0) return new_cpu; @@ -12554,13 +12551,15 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd env->fbq_type = fbq_classify_group(&sds->busiest_stat); if (!env->sd->parent) { + struct root_domain *rd = rcu_dereference(env->dst_rq->rd); + /* update overload indicator if we are at root domain */ - set_rd_overloaded(env->dst_rq->rd, sg_overloaded); + set_rd_overloaded(rd, sg_overloaded); /* Update over-utilization (tipping point, U >= 0) indicator */ - set_rd_overutilized(env->dst_rq->rd, sg_overutilized); + set_rd_overutilized(rd, sg_overutilized); } else if (sg_overutilized) { - set_rd_overutilized(env->dst_rq->rd, sg_overutilized); + set_rd_overutilized(rcu_dereference(env->dst_rq->rd), sg_overutilized); } update_idle_cpu_scan(env, sum_util); @@ -12806,8 +12805,10 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env) if (busiest->group_type == group_misfit_task) goto force_balance; - if (!is_rd_overutilized(env->dst_rq->rd) && - rcu_dereference_all(env->dst_rq->rd->pd)) + struct root_domain *rd = rcu_dereference(env->dst_rq->rd); + + if (rd && !is_rd_overutilized(rd) && + rcu_dereference_all(rd->pd)) goto out_balanced; /* ASYM feature bypasses nice load balance check */ @@ -14386,7 +14387,7 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf) if (!sd) goto out; - if (!get_rd_overloaded(this_rq->rd) || + if (!get_rd_overloaded(rcu_dereference(this_rq->rd)) || this_rq->avg_idle < sd->max_newidle_lb_cost) { update_next_balance(sd, &next_balance); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56acf502ba26..aca352e2f4a8 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1253,7 +1253,7 @@ struct rq { int membarrier_state; #endif - struct root_domain *rd; + struct root_domain __rcu *rd; struct sched_domain __rcu *sd; struct balance_callback *balance_callback; -- 2.55.0