From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00A73C3DA6D for ; Mon, 19 May 2025 06:32:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C37FE6B008C; Mon, 19 May 2025 02:32:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE6E26B0092; Mon, 19 May 2025 02:32:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAFC26B0093; Mon, 19 May 2025 02:32:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8B4106B008C for ; Mon, 19 May 2025 02:32:19 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CE9E41A12EC for ; Mon, 19 May 2025 06:32:23 +0000 (UTC) X-FDA: 83458688166.12.770C539 Received: from out-173.mta1.migadu.com (out-173.mta1.migadu.com [95.215.58.173]) by imf26.hostedemail.com (Postfix) with ESMTP id 3375E140007 for ; Mon, 19 May 2025 06:32:21 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=YTR+k8xZ; spf=pass (imf26.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.173 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747636342; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NI3xirp7ao0Fx3rWIO8hGtK9meMvvQYF3ylZpIHjujo=; b=bug1w9EosCXn1pQIa0GOACvaCN6ADJFevCRAx4LrMVDrmTQp9g6AKGWbB9W2RShi/1SmHU 8JmaHZjgmOyq/2wrLJd8+gJGpzemqF01whQyTYxo4cgQeVwyGdWBtMI5nujEa+0KqLzGrQ LLxHB6YEfwH3H7PvgnTzo9sMasVJp5k= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=YTR+k8xZ; spf=pass (imf26.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.173 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747636342; a=rsa-sha256; cv=none; b=Dz706+4VofKPGJ3R5QPbk6g220BiwJm9nHUKr4U0vkue5x2IWKz0bEvuT2ezI8Xbmv8GAw YjlucdY5WFSxVanWa7dp6kXOMNjLCDNXXFG5JvY/PZAL2EY8l7orGtUIgDIJB7kbde+Hf+ XoCYjdKCM3ga+T9rd9UW8w7aCxSSH50= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1747636338; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NI3xirp7ao0Fx3rWIO8hGtK9meMvvQYF3ylZpIHjujo=; b=YTR+k8xZ05MCc/6JcE3uYiLGWmAbv1cuLzYsqzq439kB4VxbzUeZk55mBbbNYbtjXtxmpY qMw+AozgxRFQqB0Aya1A1XnGEMp8N0A0a7Wp+nf6zAR2ZXPEwYjLBCV6GA+8nSb1EJdsDm hYzeDputBXbEylRKfHiOw5c2v/fvrFE= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , Harry Yoo , Yosry Ahmed , Peter Zijlstra , Mathieu Desnoyers , Tejun Heo , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v4 2/5] memcg: nmi safe memcg stats for specific archs Date: Sun, 18 May 2025 23:31:39 -0700 Message-ID: <20250519063142.111219-3-shakeel.butt@linux.dev> In-Reply-To: <20250519063142.111219-1-shakeel.butt@linux.dev> References: <20250519063142.111219-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam10 X-Stat-Signature: rwe8krsyfhdajxugutux1is81ezy3tis X-Rspamd-Queue-Id: 3375E140007 X-Rspam-User: X-HE-Tag: 1747636341-981955 X-HE-Meta: U2FsdGVkX18Wh/sPPLrLUqHD3UDZYEY63d4Au79Hw/BkSdUho31LcD3HwwjKDtmQ4O58j4jh3KWSwGLwvXMguX+D5AJiHIJfBsrbS8HCeBQqMk4bhKhSzm/eE80XFrXPtZ8Q6HXpL9p97D4G3Epd/fdEL5yWI+INxAy0XSAiRLVcyevMo5syZFZznha/htRfFlVHXjKNqlWqGO/Zy6Tz4VBaZ8pUlnaYLg3nVJgJ7HJ0CsLySveD0ugEk+sZFCXoXZiCi3OTWI/Thm7l0UQbyQm8uMn5ahtCwPoLRKA78kBsKf8aXXZnIweYiaZQjYB8bMIu+dNl/EVuzCdIMM2Z4SoXiaOSTWVSU2Jxtt/RQlFtV8ysK2Rw5cR3TK87R9lQhp9pnQk+TrJ87W6/D3WHwFE+f7urh5WxLVbmjYYNSrJaLKFwCtCm1gfwY2hAAjpvh6IyRKKXuOOXFmW7zm+RfcNBwJdY+JyLt44EN+rrLk0z14/2zGNzx6VN+2r3m0AdHqf3Z7z/D4eHxAqPG4WhHjBxSfZsjvKVKaboGkFY2iRrbJgihm9O5n/vYXLlLLjr28fLt02kZUOZsTtJX21zesoa/Wa40mSsUKFDX589oTo8jU7Fo3RtvcNnn66QR7paONSc9pyuqBID8s0MNlJBFyoKDG7CMXE8ZXP8igQbU1nmKoT97Q0D2Tqku7/jGXHUO9pn1MRKI1afS8D44ojXn5Cm1H1iEc/v06Ykdw2A1oW1sxEZPBzNFhB8PEqI59VRLrTItd74cdlrtd8025VfLBatbkIfH29Dm+NB2IcRAJNSXSgp8SM5Jy98S/vFaazq/tWC9/yw0x9xvjfh+AHtLj3DQeaL2B12i96wFViraEa3YP0NPQyESshtRx54bQbBklYKO7Krgw0OBh2ic/AD+YgbZOUVAj9s X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are archs which have NMI but does not support this_cpu_* ops safely in the nmi context but they support safe atomic ops in nmi context. For such archs, let's add infra to use atomic ops for the memcg stats which can be updated in nmi. At the moment, the memcg stats which get updated in the objcg charging path are MEMCG_KMEM, NR_SLAB_RECLAIMABLE_B & NR_SLAB_UNRECLAIMABLE_B. Rather than adding support for all memcg stats to be nmi safe, let's just add infra to make these three stats nmi safe which this patch is doing. Signed-off-by: Shakeel Butt Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 10 ++++++++ init/Kconfig | 7 ++++++ mm/memcontrol.c | 49 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index f7848f73f41c..87b6688f124a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -113,6 +113,12 @@ struct mem_cgroup_per_node { CACHELINE_PADDING(_pad2_); unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; struct mem_cgroup_reclaim_iter iter; + +#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC + /* slab stats for nmi context */ + atomic_t slab_reclaimable; + atomic_t slab_unreclaimable; +#endif }; struct mem_cgroup_threshold { @@ -236,6 +242,10 @@ struct mem_cgroup { atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS]; atomic_long_t memory_events_local[MEMCG_NR_MEMORY_EVENTS]; +#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC + /* MEMCG_KMEM for nmi context */ + atomic_t kmem_stat; +#endif /* * Hint of reclaim pressure for socket memroy management. Note * that this indicator should NOT be used in legacy cgroup mode diff --git a/init/Kconfig b/init/Kconfig index a2aa49cfb8bd..e9f47baa34e0 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1013,6 +1013,13 @@ config MEMCG_NMI_UNSAFE depends on !ARCH_HAS_NMI_SAFE_THIS_CPU_OPS && !ARCH_HAVE_NMI_SAFE_CMPXCHG default y +config MEMCG_NMI_SAFETY_REQUIRES_ATOMIC + bool + depends on MEMCG + depends on HAVE_NMI + depends on !ARCH_HAS_NMI_SAFE_THIS_CPU_OPS && ARCH_HAVE_NMI_SAFE_CMPXCHG + default y + config MEMCG_V1 bool "Legacy cgroup v1 memory controller" depends on MEMCG diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 532e2c06ea60..e3f0b298308a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3967,6 +3967,53 @@ static void mem_cgroup_stat_aggregate(struct aggregate_control *ac) } } +#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC +static void flush_nmi_stats(struct mem_cgroup *memcg, struct mem_cgroup *parent, + int cpu) +{ + int nid; + + if (atomic_read(&memcg->kmem_stat)) { + int kmem = atomic_xchg(&memcg->kmem_stat, 0); + int index = memcg_stats_index(MEMCG_KMEM); + + memcg->vmstats->state[index] += kmem; + if (parent) + parent->vmstats->state_pending[index] += kmem; + } + + for_each_node_state(nid, N_MEMORY) { + struct mem_cgroup_per_node *pn = memcg->nodeinfo[nid]; + struct lruvec_stats *lstats = pn->lruvec_stats; + struct lruvec_stats *plstats = NULL; + + if (parent) + plstats = parent->nodeinfo[nid]->lruvec_stats; + + if (atomic_read(&pn->slab_reclaimable)) { + int slab = atomic_xchg(&pn->slab_reclaimable, 0); + int index = memcg_stats_index(NR_SLAB_RECLAIMABLE_B); + + lstats->state[index] += slab; + if (plstats) + plstats->state_pending[index] += slab; + } + if (atomic_read(&pn->slab_unreclaimable)) { + int slab = atomic_xchg(&pn->slab_unreclaimable, 0); + int index = memcg_stats_index(NR_SLAB_UNRECLAIMABLE_B); + + lstats->state[index] += slab; + if (plstats) + plstats->state_pending[index] += slab; + } + } +} +#else +static void flush_nmi_stats(struct mem_cgroup *memcg, struct mem_cgroup *parent, + int cpu) +{} +#endif + static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); @@ -3975,6 +4022,8 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) struct aggregate_control ac; int nid; + flush_nmi_stats(memcg, parent, cpu); + statc = per_cpu_ptr(memcg->vmstats_percpu, cpu); ac = (struct aggregate_control) { -- 2.47.1