From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87E46C71136 for ; Tue, 17 Jun 2025 19:57:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 261676B0098; Tue, 17 Jun 2025 15:57:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 212466B009F; Tue, 17 Jun 2025 15:57:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B19B6B00A1; Tue, 17 Jun 2025 15:57:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id ED9C36B0098 for ; Tue, 17 Jun 2025 15:57:42 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 96667C0CC9 for ; Tue, 17 Jun 2025 19:57:42 +0000 (UTC) X-FDA: 83565952764.14.C384FAC Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by imf13.hostedemail.com (Postfix) with ESMTP id BE01F20004 for ; Tue, 17 Jun 2025 19:57:40 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ksYBtdU5; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750190260; a=rsa-sha256; cv=none; b=kAaBOzByXdgJQn3nqEYXu+uSe+i6j4AF4UJxcVZLCXgIsid+Viebj7Fff0xmdKucFkzXPv Pxf2Cv61n6kM5r7sbsGWLikAsC0HdgMgxudD1z4zczP80Xf95F6UNqkdml+DuwbbLG7u+z QClImRzx9QIlCi/e1UqbKo2BS8ggl4Q= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ksYBtdU5; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750190260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RuXBCpwuDEI0gqG4HUctbQ7ZAR2IU1ANbbo6C4hyxvo=; b=4siWpecuqRaWytST4TneyZtiYAAyjbVfT+LXB/psk1VjPAt6gtOABqd3xyLSPFlrCntQV9 GI9rKkM+NPBHUVRHQhm/fUsL8QbznY4qt+wbp7BjR1pjsoROCRrPcEzbNyeXpMwRfXIgB5 v7U+uUqNjOtbOxwZmtmjOyUWsMr34mU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1750190258; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RuXBCpwuDEI0gqG4HUctbQ7ZAR2IU1ANbbo6C4hyxvo=; b=ksYBtdU5CySMUYVnlyVTTVB+k7YYn0LsOFueTOnAykFrJq8FN4z8nqHvECIwRAEtOW5XEr 5XxtFhBRhDBWf7WbEo6G0fiCNIk0pncH2ltXaZYwNEhWbcYHtgJxIFqpAtDS6A32etGpnb ciR4v0kYykgAkQgA+dpsvu8l69RjB84= From: Shakeel Butt To: Tejun Heo Cc: Andrew Morton , JP Kobryn , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v3 1/4] cgroup: support to enable nmi-safe css_rstat_updated Date: Tue, 17 Jun 2025 12:57:22 -0700 Message-ID: <20250617195725.1191132-2-shakeel.butt@linux.dev> In-Reply-To: <20250617195725.1191132-1-shakeel.butt@linux.dev> References: <20250617195725.1191132-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: BE01F20004 X-Stat-Signature: zue5yho1ecbubxixkd33bzu3pnu4pn6g X-Rspam-User: X-HE-Tag: 1750190260-804303 X-HE-Meta: U2FsdGVkX1/j8PO34tmzhVGm4TfSuJoKc5Bf9NsPiRdvGR1RR6AZ1yoLmj6OZ4Mq9WwjQUK62QGJBPKpgVvwd13T3uycZbYvQknoUa8ppYVzShF+Y8VtwqzNKaJnzj6a+GK+b85KYQmcRM1b2+0+GCVkw3dqF/ObE3XslSJ0esVQ4u5CjvTUOzxlfVb/5N6HQPgA/LnvYJ0piO+P1mpFPcruQxNwT0kgM0tZMKXprplnUgreb2i0cpMNtTOHRD8nYqJHFnokP9fF/Yuj1RRiS6kCOTm1Wk7esjky/qBLoKu814KmEaPqeDLSln4yuD/0yQGvHlzUAM8yXHaJzeLr+Jdi5lelXZLtQFn9DUCUH+DyWOxL/bVvn6noGvv5s0C0XLi+u3fXrN1bwexXAqwcgZePoichj5MYlx1H1SQfU8InbFtFI0wTuVPyu7kiy1kruAZtI4zUFnacikFWWWTwfyotvGZFNA/AhgQD2C3ODbsIilPbQowMVjGRhp6wiosIseumDaGCuBtV5B8G7nGiHwug9Pu64N3tQlpdyUDshTGgxVYpzo2IhuA8CDcSID9zpQziuzFdAD2p5ghFdNUU1vPhzfr0on8ihRgFPq0bcOrV67ZPKe68i4Hkkzn3qDWGjNBvX2jFwC/PJYQI7RACZKB/a5rD3ypqAob7eNi4Nlqv1eOqarM/baoEHp3jBjCxOV8JN8lVa8iGOaP+JF2TIkd932h4HunPQPOLHwqhxYToJ0ck4O6h/R2P+IjjMPMj1zQCy7z628L/4+3TtHPOhfbLkFNjByYW8H9H7CfGUjVyPdDONgbLKYSbSn44kwn0vv006j39/CKJxvsf50mYGI8a2Oss1Ti2VLF7bZ97NLZ5GVAkQxaXmKHTEK8fchxrEAXcMCV7YiwfWX80tr5u8T2P8w6+BooT5e1hFAjahvmyq02Wx18AL9UllvpAPwFCLtTlfiITJSXDn49zO+Y khVzpxKV 1kwU/w7gTtCivG0MHnRpECynyH0P6OVDsOXThgDZhIFE2UENIBekL+noJR2aFpwP8GD4ekFLiz8x958nAk4leS640ADiKInKnxdv4y2YU8sysn5GeRECOgbPK0lJiVNp60w20FReL4sdUqaoYWcwlAza+rC7mQD0rroQ9oJUNbgEsPQWFv8X912NdLNC5LKf0fnPOvMFCLUVy5CIou6Q+cK+0vkdrVM7tpSIib+JVtIPMS6YN4nV4KFqKlVs++R1VtVeG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add necessary infrastructure to enable the nmi-safe execution of css_rstat_updated(). Currently css_rstat_updated() takes a per-cpu per-css raw spinlock to add the given css in the per-cpu per-css update tree. However the kernel can not spin in nmi context, so we need to remove the spinning on the raw spinlock in css_rstat_updated(). To support lockless css_rstat_updated(), let's add necessary data structures in the css and ss structures. Signed-off-by: Shakeel Butt --- include/linux/cgroup-defs.h | 4 ++++ kernel/cgroup/rstat.c | 23 +++++++++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index cd7f093e34cd..04191d99228c 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -384,6 +384,9 @@ struct css_rstat_cpu { */ struct cgroup_subsys_state *updated_children; struct cgroup_subsys_state *updated_next; /* NULL if not on the list */ + + struct llist_node lnode; /* lockless list for update */ + struct cgroup_subsys_state *owner; /* back pointer */ }; /* @@ -822,6 +825,7 @@ struct cgroup_subsys { spinlock_t rstat_ss_lock; raw_spinlock_t __percpu *rstat_ss_cpu_lock; + struct llist_head __percpu *lhead; /* lockless update list head */ }; extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index ce4752ab9e09..bfa6366d2325 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -11,6 +11,7 @@ static DEFINE_SPINLOCK(rstat_base_lock); static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock); +static DEFINE_PER_CPU(struct llist_head, rstat_backlog_list); static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); @@ -45,6 +46,13 @@ static spinlock_t *ss_rstat_lock(struct cgroup_subsys *ss) return &rstat_base_lock; } +static inline struct llist_head *ss_lhead_cpu(struct cgroup_subsys *ss, int cpu) +{ + if (ss) + return per_cpu_ptr(ss->lhead, cpu); + return per_cpu_ptr(&rstat_backlog_list, cpu); +} + static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu) { if (ss) @@ -456,7 +464,8 @@ int css_rstat_init(struct cgroup_subsys_state *css) for_each_possible_cpu(cpu) { struct css_rstat_cpu *rstatc = css_rstat_cpu(css, cpu); - rstatc->updated_children = css; + rstatc->owner = rstatc->updated_children = css; + init_llist_node(&rstatc->lnode); if (is_self) { struct cgroup_rstat_base_cpu *rstatbc; @@ -525,9 +534,19 @@ int __init ss_rstat_init(struct cgroup_subsys *ss) } #endif + if (ss) { + ss->lhead = alloc_percpu(struct llist_head); + if (!ss->lhead) { + free_percpu(ss->rstat_ss_cpu_lock); + return -ENOMEM; + } + } + spin_lock_init(ss_rstat_lock(ss)); - for_each_possible_cpu(cpu) + for_each_possible_cpu(cpu) { raw_spin_lock_init(ss_rstat_cpu_lock(ss, cpu)); + init_llist_head(ss_lhead_cpu(ss, cpu)); + } return 0; } -- 2.47.1