From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25D83C71135 for ; Wed, 11 Jun 2025 22:16:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B63D76B008A; Wed, 11 Jun 2025 18:16:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AED866B0092; Wed, 11 Jun 2025 18:16:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2A806B0093; Wed, 11 Jun 2025 18:16:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7F1066B008A for ; Wed, 11 Jun 2025 18:16:04 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3BE441216CE for ; Wed, 11 Jun 2025 22:16:04 +0000 (UTC) X-FDA: 83544528648.15.E5F8ADE Received: from out-172.mta0.migadu.com (out-172.mta0.migadu.com [91.218.175.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 95286180003 for ; Wed, 11 Jun 2025 22:16:02 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=lmvnBckx; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.172 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749680162; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vjwAclPLPNB0cOL9u2mwbYWPRL/XnTNpPecdd20u+W8=; b=nAfSFL1fe1VfIec4IA9umRzV1q6CFknkGo27KAGh+gl0LMefhobVQzedkLN1FybKEOftgJ wJoc1OF/Twx1nV4UIR0WoAhzvNw1JBs6J97Iz3hkiNBUqMsM8kmcSO+YD25dlFn3tq89QV YocnRO4L3qi72VZIrGiaN8YsW4KmrZY= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=lmvnBckx; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.172 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749680162; a=rsa-sha256; cv=none; b=J5ImwIeypC2pW2QBYJ+uWEbwUjsT3mHM1EgqB287oEiIq6WVSz2vqYxvnW2srVIO6whG/4 lVpwEvizUDCVSIi4MWEFJMw6VllrIrFUE80+3lt0mCYFO2PQpFbdgG5MKb6vh0EBnLDRtK UROuK9nAfynPFc+IWPIGglJPTzuWcuk= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1749680160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vjwAclPLPNB0cOL9u2mwbYWPRL/XnTNpPecdd20u+W8=; b=lmvnBckx5QukiFLgXOXw6Ddt0Ee9rJzer/x35eKhcwsjY3akw+jLLRISXc5Go7csrKE2hJ Gv2I3QBXukt/xjHNHnTa2/tAbrQMv5cQVeENziG39GtwDnLti7Wdgb2hvQ0n7dWUxxznN/ iNIh7XjN2aO+x7TaKCMu/LqerSxxep8= From: Shakeel Butt To: Tejun Heo , Andrew Morton Cc: JP Kobryn , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2 1/4] cgroup: support to enable nmi-safe css_rstat_updated Date: Wed, 11 Jun 2025 15:15:29 -0700 Message-ID: <20250611221532.2513772-2-shakeel.butt@linux.dev> In-Reply-To: <20250611221532.2513772-1-shakeel.butt@linux.dev> References: <20250611221532.2513772-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 95286180003 X-Stat-Signature: om5qbi3ca4cjjs8374hgfk1qm6osxfrq X-Rspam-User: X-HE-Tag: 1749680162-117527 X-HE-Meta: U2FsdGVkX1+w8m1W5kS3RdEYum+5pwifEbHwy/qTbL/96P2LMZX2sAMyrv7y5Mzqd+yEJwvFBPzCvIEE0WT84dqToQjiH7/NKeCe59mAYcc4w3TA93KsYW6fPqamNcRbSpfmLpI7qmFbjVE3UPTntziafGT3mUz0FTDLwb0NkvqGk3K1V6dGz9WcXXavSYx+M9IJmFI/2n+i5LhJ46u96KAI0s2q16uph/INT7gZ6cDgUSN7nbGa/ss8Bl/DDIK8gzBZ9vNtabQYlA9D/Ou5zDli1AyWGEUlMVMokFREpPSacenemovB9/Q6F4E7R6nP1R8gfoYl3aE7NRHTdeuu6nVoMtdT1AFX5dCFs88L+DIg3KQ97cT8SGWQ0fS07KDUveMKlUyM67yHWWmVs2ccxOEjz9x6w1yPm4ZJJKKy7+b/XNSx/XSE9QlBeOh3Nx+HW1GFD7fkgBbnPWqexMM/O9B55cSSmrZROhMMQnawXp5pcHW7BOovZ+m6Bi/pBYDrXnUy31FLFsTaxD8uR8oXMBTh3f/6wZxWtQXlz7xOcnDGr+JhFUxrLfaKrFMzWmNOXMTr4/AWZJWq1+W6fUx3x4kkNvVIB77q7oHPqC0NGLhLRITWRN8OkxEVCFbamH4A7kYnncPEApX0skOPE1Inh2dKJXyx64HB6uZvvpApIPvTCsJDuxWD05A1Ll3+tydXIKBq6RbIB9gnHVeiSdBSpf434IGDqIBKr8Cq5eL7CoDQb2yKRQf00uCi0eUON9bSfW0Uc9X6mvVe9zN5rrnKJxvDuZesE7zkEJUD5rdwxNmqIdsXDChBL+3K2KWFGVXpFeZMK4dszHr65waHyW80SstRawDNNL8H3ugnQhVjTU5wkpBleslyS+uvPTyvcyuci1Eo2xNPTU8b81qcI6wL+jkilFKVf8a3rk1uQdjbq9XxERk/xsBAyDiVn1WUfPzjxEYaLni36NzXRU1e92V ez+TEXsw glCTV4uphqJ/Pjd7ti6L9jpidl1Cd8zDBCHXMBg7d7NadHPCBRqVld0t+PR1lXGR6cOvPwXJmJA0nCXQp81WgxKxh0GF7XSIqEFutpOwiu+W7c/wqRzQMtptBuSJ8jbMl1Fd25dgrmMHJG89SReXyXAKm4lCDV7/MemzyIVloPdsoe3LLPvrHLFBv5Sx5PcUAdeLwuJjD+9kr8L3TLvb7qkH+lmwzkZneSFBRH+58OTGY4U1PwjN2GkLvvoQFKl8UoScW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add necessary infrastructure to enable the nmi-safe execution of css_rstat_updated(). Currently css_rstat_updated() takes a per-cpu per-css raw spinlock to add the given css in the per-cpu per-css update tree. However the kernel can not spin in nmi context, so we need to remove the spinning on the raw spinlock in css_rstat_updated(). To support lockless css_rstat_updated(), let's add necessary data structures in the css and ss structures. Signed-off-by: Shakeel Butt --- include/linux/cgroup-defs.h | 4 ++++ kernel/cgroup/rstat.c | 23 +++++++++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index e61687d5e496..45860fe5dd0c 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -384,6 +384,9 @@ struct css_rstat_cpu { */ struct cgroup_subsys_state *updated_children; struct cgroup_subsys_state *updated_next; /* NULL if not on the list */ + + struct llist_node lnode; /* lockless list for update */ + struct cgroup_subsys_state *owner; /* back pointer */ }; /* @@ -822,6 +825,7 @@ struct cgroup_subsys { spinlock_t rstat_ss_lock; raw_spinlock_t __percpu *rstat_ss_cpu_lock; + struct llist_head __percpu *lhead; /* lockless update list head */ }; extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index cbeaa499a96a..a5608ae2be27 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -11,6 +11,7 @@ static DEFINE_SPINLOCK(rstat_base_lock); static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock); +static DEFINE_PER_CPU(struct llist_head, rstat_backlog_list); static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); @@ -45,6 +46,13 @@ static spinlock_t *ss_rstat_lock(struct cgroup_subsys *ss) return &rstat_base_lock; } +static inline struct llist_head *ss_lhead_cpu(struct cgroup_subsys *ss, int cpu) +{ + if (ss) + return per_cpu_ptr(ss->lhead, cpu); + return per_cpu_ptr(&rstat_backlog_list, cpu); +} + static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu) { if (ss) { @@ -468,7 +476,8 @@ int css_rstat_init(struct cgroup_subsys_state *css) for_each_possible_cpu(cpu) { struct css_rstat_cpu *rstatc = css_rstat_cpu(css, cpu); - rstatc->updated_children = css; + rstatc->owner = rstatc->updated_children = css; + init_llist_node(&rstatc->lnode); if (is_self) { struct cgroup_rstat_base_cpu *rstatbc; @@ -532,9 +541,19 @@ int __init ss_rstat_init(struct cgroup_subsys *ss) return -ENOMEM; } + if (ss) { + ss->lhead = alloc_percpu(struct llist_head); + if (!ss->lhead) { + free_percpu(ss->rstat_ss_cpu_lock); + return -ENOMEM; + } + } + spin_lock_init(ss_rstat_lock(ss)); - for_each_possible_cpu(cpu) + for_each_possible_cpu(cpu) { raw_spin_lock_init(ss_rstat_cpu_lock(ss, cpu)); + init_llist_head(ss_lhead_cpu(ss, cpu)); + } return 0; } -- 2.47.1