From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D61BDC369DC for ; Tue, 29 Apr 2025 06:12:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB9A76B000D; Tue, 29 Apr 2025 02:12:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E68976B000E; Tue, 29 Apr 2025 02:12:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8B4D6B0011; Tue, 29 Apr 2025 02:12:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BACF66B000D for ; Tue, 29 Apr 2025 02:12:50 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AE81FBADF7 for ; Tue, 29 Apr 2025 06:12:50 +0000 (UTC) X-FDA: 83386062900.02.74D295B Received: from out-170.mta1.migadu.com (out-170.mta1.migadu.com [95.215.58.170]) by imf13.hostedemail.com (Postfix) with ESMTP id 178012000E for ; Tue, 29 Apr 2025 06:12:48 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=kUP0FEeY; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.170 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745907169; a=rsa-sha256; cv=none; b=cxwW6+AC6I50TAJ25csXCgrIQ0bqfyDLZpZDQ1v3bnyTnzeGGG/FSN2N7N4B7ON9KDt7V4 ioh7nVYS/ggCCU5avg4Kx2eW9bgUL2KocwM7eUTLlNvmE4y0RFw6gNO7VLozBFg4ud2Zzp W02qBM2/LJaBOD5bQ/PeeIMFiMULAtU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=kUP0FEeY; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.170 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745907169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Pg3Smiuwje0/RTT5Q+S0BzKIdA3RdCTOLfpJO4h8ixE=; b=SQuCyyKioH3LFm4RdwCCkydBciejw2I73F2MNHNIX2yJ5L+LuuKaWXO+flQc5kt0ZeuJ9i 8/Pzdsc9+dtn8yhAhHjL5owEiPCADDvEBXZkFcUwsgs4chbknU0cPbIVGK6mdKaPmc6qv7 T1lhr+vTmklmtMxVzob1uoHksS1K+Gw= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1745907167; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Pg3Smiuwje0/RTT5Q+S0BzKIdA3RdCTOLfpJO4h8ixE=; b=kUP0FEeYUvU+JxCiLGd/dlh+iU8tqDznw0ETGbp/E29HWv5bRwsmEEAKMneXxpnpPeUhGg p93vTnCI7eVJDuOaXpOFHjtXufQ+ZvRyx4lAmIUSJFznRpuX/hwFd77HSLlAl2qT2pDIWM G0qwvMpA+/51AI0epgL6067yCI1jtWU= From: Shakeel Butt To: Tejun Heo , Andrew Morton , Alexei Starovoitov Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Yosry Ahmed , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Vlastimil Babka , Sebastian Andrzej Siewior , JP Kobryn , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [RFC PATCH 2/3] cgroup: support to enable nmi-safe css_rstat_updated Date: Mon, 28 Apr 2025 23:12:08 -0700 Message-ID: <20250429061211.1295443-3-shakeel.butt@linux.dev> In-Reply-To: <20250429061211.1295443-1-shakeel.butt@linux.dev> References: <20250429061211.1295443-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 178012000E X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: atrd5axnonyo9ebkwuywprijfqqdcakm X-HE-Tag: 1745907168-181934 X-HE-Meta: U2FsdGVkX1/wmQD6Td5ZNpTWY2teK4CPMcdYVAgbuY6srkryA0FTN1WB5mhXA0vqwe3piUk25nWrVeMBEa+vEyB/bMIsN8Al5WzERb+qsu4fV/BMuJJvJv8ia0UysSUvCs7aBRIWqXmtocTxeJvaWZ++58jvqNkFB5JUjLTW8jL4IENkj0WOAGNO+f/Klt/A92LB0P2WMaUjev1IZtYoIa7FQnfQ6wFk7itcFrijLQQShdItypl6AA14V8Q/5Iww8Dh5jEd6RaRRZ2dcXtYtWxOVl5Gxx8Fr3+k9jrscFNaBfRRVXxboRp1f/NVwiMLWooElJ6Q+P6gyBoWlkXjTAVgSdoKEgQvn42hMRyFg6Up/S8/veqEMu+eAJyqrYfBOCLsD2W6TFaovxqOT5wSGhgmG68Z1MY9ZU/ApoDeUi9ojllS4nMbuOu7ludtoSPOvutT3cx8ezLukyLs14I6156C2vbV9pnyFJnFTOZTnFpmBFkYvr+CxeCVxUpcyvqzL3zHhjATeiZ8v49aObx37T74aUKd5+w9TH5DijKHuH6hJgCXzG9w03AzpXiWbx2YS2oNQlhqc44134UMYl3Jnu/WIQ9j60Blt95CY1r7V7gyR2z43HeRPiJH7ioKukz24emo7NZ56PAdQ/m4ytyK3Gcvc8zkooFI4zO4PYXl49k/KrCBlRMBK4w231hU5/0ba/lATSPrTzy70ljjmQV75F0Mldv4Nuv9ERy+gSRhbPck9Zv47aukvC6dJfYyKwcwJNzB9PvDLG7+hx0JmhjNcXLFmZ7NbTUnmq/NlXhrX2eJDm188LK0CnGiTfdaBbxew8jshfhJsr3jx5LDu6vqTCqYE2x+pugKwhD0TyOsbS0slsNTV9KFvUn4GBRen7gPfiUhVra7KZIBavviE1JpGaAtkrrpZ89f5RmkIO+LUYdLOCGIXevSNK3FR4RtmLRMgPgMwvg5Uw4t8GpigVSx wUtCxrZ0 gfn3pmOSRrMu+osUtmC9gu8aHKK3mi4mHJiuRCX1kmaQ3vB08t5hMDuxtmnT3Qk+PRlRv/TfmyuIuaUg/3pLau5Z6hEPrEWQvF3DtG40Df5wlQAxJf3GQSVZcbZyftaue8mbVGyKDJwuJBd9yb8UBsKR64dAuhrwlE47uOzG0BSyDXVIh/q0jL4rLHqj9RJabZeUrM62U8QGD06f3Y2W6LMS7LRyRJRUO94/v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add necessary infrastructure to enable the nmi-safe execution of css_rstat_updated(). Currently css_rstat_updated() takes a per-cpu per-css raw spinlock to add the given css in the per-cpu per-css update tree. However the kernel can not spin in nmi context, so we need to replace spinning on the raw spinlock with the trylock and on failure, add the given css to the per-cpu backlog which will be processed when the context that can spin on raw spinlock can run. For now, this patch just adds necessary data structures in the css and ss structures. Signed-off-by: Shakeel Butt --- include/linux/cgroup-defs.h | 4 ++++ kernel/cgroup/rstat.c | 13 +++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 560582c4dbeb..f7b680f853ea 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -370,6 +370,9 @@ struct css_rstat_cpu { */ struct cgroup_subsys_state *updated_children; /* terminated by self cgroup */ struct cgroup_subsys_state *updated_next; /* NULL iff not on the list */ + + struct llist_node lnode; /* lockless backlog node */ + struct cgroup_subsys_state *owner; /* back pointer */ }; /* @@ -800,6 +803,7 @@ struct cgroup_subsys { spinlock_t rstat_ss_lock; raw_spinlock_t __percpu *rstat_ss_cpu_lock; + struct llist_head __percpu *lhead; /* lockless backlog list */ }; extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index a30bcc4d4f48..d3092b4c85d7 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -419,7 +419,8 @@ int css_rstat_init(struct cgroup_subsys_state *css) for_each_possible_cpu(cpu) { struct css_rstat_cpu *rstatc = css_rstat_cpu(css, cpu); - rstatc->updated_children = css; + rstatc->owner = rstatc->updated_children = css; + init_llist_node(&rstatc->lnode); if (css_is_cgroup(css)) { struct cgroup_rstat_base_cpu *rstatbc; @@ -484,8 +485,16 @@ int __init ss_rstat_init(struct cgroup_subsys *ss) if (!ss->rstat_ss_cpu_lock) return -ENOMEM; - for_each_possible_cpu(cpu) + ss->lhead = alloc_percpu(struct llist_head); + if (!ss->lhead) { + free_percpu(ss->rstat_ss_cpu_lock); + return -ENOMEM; + } + + for_each_possible_cpu(cpu) { raw_spin_lock_init(per_cpu_ptr(ss->rstat_ss_cpu_lock, cpu)); + init_llist_head(per_cpu_ptr(ss->lhead, cpu)); + } return 0; } -- 2.47.1