* Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Barry Song @ 2026-05-16 8:39 UTC (permalink / raw)
To: T.J. Mercier
Cc: Christian König, Albert Esteve, Tejun Heo, Johannes Weiner,
Michal Koutný, Jonathan Corbet, Shuah Khan, Sumit Semwal,
Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, Benjamin Gaignard, Brian Starkey, John Stultz,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan, cgroups, linux-doc,
linux-kernel, linux-media, dri-devel, linaro-mm-sig, linux-mm,
linux-security-module, selinux, linux-kselftest, mripard,
echanude
In-Reply-To: <CABdmKX2uwZ12kYJYPJGfWxuMBOJS=64b1GRj72tfB5D=NKM22w@mail.gmail.com>
On Wed, May 13, 2026 at 2:54 AM T.J. Mercier <tjmercier@google.com> wrote:
>
> On Tue, May 12, 2026 at 3:14 AM Christian König
> <christian.koenig@amd.com> wrote:
> >
> > On 5/12/26 11:10, Albert Esteve wrote:
> > > On embedded platforms a central process often allocates dma-buf
> > > memory on behalf of client applications. Without a way to
> > > attribute the charge to the requesting client's cgroup, the
> > > cost lands on the allocator, making per-cgroup memory limits
> > > ineffective for the actual consumers.
> > >
> > > Add charge_pid_fd to struct dma_heap_allocation_data. When set to
> > > a valid pidfd, DMA_HEAP_IOCTL_ALLOC resolves the target task's
> > > memcg and charges the buffer there via mem_cgroup_charge_dmabuf()
> > > inside dma_heap_buffer_alloc(). Without charge_pid_fd, and with
> > > the mem_accounting module parameter enabled, the buffer is charged
> > > to the allocator's own cgroup.
> > >
> > > Additionally, commit 3c227be90659 ("dma-buf: system_heap: account for
> > > system heap allocation in memcg") adds __GFP_ACCOUNT to system-heap
> > > page allocations. Keeping __GFP_ACCOUNT would charge the same pages
> > > twice (once to kmem, once to MEMCG_DMABUF), thus remove it and route
> > > all accounting through a single MEMCG_DMABUF path.
> > >
> > > Usage examples:
> > >
> > > 1. Central allocator charging to a client at allocation time.
> > > The allocator knows the client's PID (e.g., from binder's
> > > sender_pid) and uses pidfd to attribute the charge:
> > >
> > > pid_t client_pid = txn->sender_pid;
> > > int pidfd = pidfd_open(client_pid, 0);
> > >
> > > struct dma_heap_allocation_data alloc = {
> > > .len = buffer_size,
> > > .fd_flags = O_RDWR | O_CLOEXEC,
> > > .charge_pid_fd = pidfd,
> > > };
> > > ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> > > close(pidfd);
> > > /* alloc.fd is now charged to client's cgroup */
> > >
> > > 2. Default allocation (no pidfd, mem_accounting=1).
> > > When charge_pid_fd is not set and the mem_accounting module
> > > parameter is enabled, the buffer is charged to the allocator's
> > > own cgroup:
> > >
> > > struct dma_heap_allocation_data alloc = {
> > > .len = buffer_size,
> > > .fd_flags = O_RDWR | O_CLOEXEC,
> > > };
> > > ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> > > /* charged to current process's cgroup */
> > >
> > > Current limitations:
> > >
> > > - Single-owner model: a dma-buf carries one memcg charge regardless of
> > > how many processes share it. Means only the first owner (and exporter)
> > > of the shared buffer bears the charge.
> > > - Only memcg accounting supported. While this makes sense for system
> > > heap buffers, other heaps (e.g., CMA heaps) will require selectively
> > > charging also for the dmem controller.
> >
> > Well that doesn't looks soo bad, it at least seems to tackle the problem at hand for Android and some of other embedded use cases.
>
> Yeah I think this might work. I know of 3 cases, and it trivially
> solves the first two. The third requires some work on our end to
> extend our userspace interfaces to include the pidfd but it seems
> doable. I'm checking with our graphics folks.
>
> 1) Direct allocation from user (e.g. app -> allocation ioctl on
> /dev/dma_heap/foo)
> No changes required to userspace. mem_accounting=1 charges the app.
>
> 2) Single hop remote allocation (e.g. app -> AHardwareBuffer_allocate
> -> gralloc)
> gralloc has the caller's pid as described in the commit message. Open
> a pidfd and pass it in the dma_heap_allocation_data.
>
> 3) Double hop remote allocation (e.g. app -> dequeueBuffer ->
> SurfaceFlinger -> gralloc)
> In this case gralloc knows SurfaceFlinger's pid, but not the app's. So
> we need to add the app's pidfd to the SurfaceFlinger -> gralloc
> interface, or transfer the memcg charge from SurfaceFlinger to the app
> after the allocation.
> It'd be nice to avoid the charge transfer option entirely, but if we
> need it that doesn't seem so bad in this case because it's a bulk
> charge for the entire dmabuf rather than per-page. So the exporter
> doesn't need to get involved (we wouldn't need a new dma_buf_op) and
> we wouldn't have to worry about looping and locking for each page.
>
Hi T.J.,
Your description of the three different cases sounds very interesting.
It helps me understand how difficult it can be to correctly charge
dma-buf in the current user scenarios.
I’m wondering where I can find Android userspace code that transfers
the PID of RPC callers. Do we have any existing sample code in Android
for this?
> > I'm just not sure if this is future prove and will work for all use cases, e.g. cloud gaming, native context for automotive etc...
Thanks
Barry
^ permalink raw reply
* Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Barry Song @ 2026-05-16 7:36 UTC (permalink / raw)
To: Albert Esteve
Cc: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan, cgroups, linux-doc,
linux-kernel, linux-media, dri-devel, linaro-mm-sig, linux-mm,
linux-security-module, selinux, linux-kselftest, mripard,
echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-2-6326701c3691@redhat.com>
On Tue, May 12, 2026 at 5:18 PM Albert Esteve <aesteve@redhat.com> wrote:
>
> On embedded platforms a central process often allocates dma-buf
> memory on behalf of client applications. Without a way to
> attribute the charge to the requesting client's cgroup, the
> cost lands on the allocator, making per-cgroup memory limits
> ineffective for the actual consumers.
>
> Add charge_pid_fd to struct dma_heap_allocation_data. When set to
> a valid pidfd, DMA_HEAP_IOCTL_ALLOC resolves the target task's
> memcg and charges the buffer there via mem_cgroup_charge_dmabuf()
> inside dma_heap_buffer_alloc(). Without charge_pid_fd, and with
> the mem_accounting module parameter enabled, the buffer is charged
> to the allocator's own cgroup.
>
> Additionally, commit 3c227be90659 ("dma-buf: system_heap: account for
> system heap allocation in memcg") adds __GFP_ACCOUNT to system-heap
> page allocations. Keeping __GFP_ACCOUNT would charge the same pages
> twice (once to kmem, once to MEMCG_DMABUF), thus remove it and route
> all accounting through a single MEMCG_DMABUF path.
>
[...]
> - if (mem_accounting)
> - flags |= __GFP_ACCOUNT;
Hi Albert,
would it be better to move this and its description to patch 1? It
looks like patch 1 already introduces the double accounting changes,
and patch 2 is mainly just supporting remote charging.
Also, mem_accounting is only used by system_heap.c; has this patchset
also eliminated its need?
Thanks
Barry
^ permalink raw reply
* [PATCH v2] cgroup/rstat: validate cpu before css_rstat_cpu() access
From: Qing Ming @ 2026-05-16 7:08 UTC (permalink / raw)
To: Tejun Heo, Josef Bacik, Jens Axboe, Johannes Weiner,
Michal Koutný, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Alexei Starovoitov, Hao Luo,
Yosry Ahmed
Cc: cgroups, linux-block, linux-kernel, linux-mm, bpf, Qing Ming
In-Reply-To: <20260515122952.59209-1-a0yami@mailbox.org>
css_rstat_updated() is exposed as a BPF kfunc and accepts a
caller-provided cpu argument. The function uses cpu for per-cpu rstat
lookups without checking whether it refers to a valid possible CPU.
A BPF iter/cgroup program with CAP_BPF and CAP_PERFMON can pass an
invalid cpu value. On an unfixed UBSCAN_BOUNDS test kernel, cpu ==
0x7fffffff triggers:
UBSAN: array-index-out-of-bounds in kernel/cgroup/rstat.c:31:9
index 2147483647 is out of range for type 'long unsigned int [64]'
Call Trace:
css_rstat_updated
bpf_iter_run_prog
cgroup_iter_seq_show
bpf_seq_read
Add cpu validation to the BPF-facing css_rstat_updated() kfunc and
move the common implementation to __css_rstat_updated() for in-kernel
callers.
Fixes: a319185be9f5 ("cgroup: bpf: enable bpf programs to integrate with rstat")
Signed-off-by: Qing Ming <a0yami@mailbox.org>
---
v2:
- Split css_rstat_updated() into a BPF-visible wrapper and an internal
__css_rstat_updated() helper.
- Switch internal callers to __css_rstat_updated().
block/blk-cgroup.c | 2 +-
include/linux/cgroup.h | 1 +
kernel/cgroup/rstat.c | 30 ++++++++++++++++++++----------
mm/memcontrol.c | 6 +++---
4 files changed, 25 insertions(+), 14 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 554c87bb4a86..bc63bd220865 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -2241,7 +2241,7 @@ void blk_cgroup_bio_start(struct bio *bio)
}
u64_stats_update_end_irqrestore(&bis->sync, flags);
- css_rstat_updated(&blkcg->css, cpu);
+ __css_rstat_updated(&blkcg->css, cpu);
put_cpu();
}
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index f6d037a30fd8..c5648fcf74e2 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -777,6 +777,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen)
/*
* cgroup scalable recursive statistics.
*/
+void __css_rstat_updated(struct cgroup_subsys_state *css, int cpu);
void css_rstat_updated(struct cgroup_subsys_state *css, int cpu);
void css_rstat_flush(struct cgroup_subsys_state *css);
diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c
index 150e5871e66f..ed60ba119c68 100644
--- a/kernel/cgroup/rstat.c
+++ b/kernel/cgroup/rstat.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-only
#include "cgroup-internal.h"
+#include <linux/cpumask.h>
#include <linux/sched/cputime.h>
#include <linux/bpf.h>
@@ -53,7 +54,7 @@ static inline struct llist_head *ss_lhead_cpu(struct cgroup_subsys *ss, int cpu)
}
/**
- * css_rstat_updated - keep track of updated rstat_cpu
+ * __css_rstat_updated - keep track of updated rstat_cpu
* @css: target cgroup subsystem state
* @cpu: cpu on which rstat_cpu was updated
*
@@ -63,20 +64,17 @@ static inline struct llist_head *ss_lhead_cpu(struct cgroup_subsys *ss, int cpu)
*
* NOTE: if the user needs the guarantee that the updater either add itself in
* the lockless list or the concurrent flusher flushes its updated stats, a
- * memory barrier is needed before the call to css_rstat_updated() i.e. a
+ * memory barrier is needed before the call to __css_rstat_updated() i.e. a
* barrier after updating the per-cpu stats and before calling
- * css_rstat_updated().
+ * __css_rstat_updated().
*/
-__bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
+void __css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
{
struct llist_head *lhead;
struct css_rstat_cpu *rstatc;
struct llist_node *self;
- /*
- * Since bpf programs can call this function, prevent access to
- * uninitialized rstat pointers.
- */
+ /* Prevent access to uninitialized rstat pointers. */
if (!css_uses_rstat(css))
return;
@@ -125,6 +123,18 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
llist_add(&rstatc->lnode, lhead);
}
+/*
+ * BPF-facing wrapper for __css_rstat_updated(). Validate the caller-provided
+ * CPU before passing it to the internal rstat updater.
+ */
+__bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
+{
+ if (unlikely(cpu < 0 || cpu >= nr_cpu_ids || !cpu_possible(cpu)))
+ return;
+
+ __css_rstat_updated(css, cpu);
+}
+
static void __css_process_update_tree(struct cgroup_subsys_state *css, int cpu)
{
/* put @css and all ancestors on the corresponding updated lists */
@@ -170,7 +180,7 @@ static void css_process_update_tree(struct cgroup_subsys *ss, int cpu)
* flusher flush the stats updated by the updater who have
* observed that they are already on the list. The
* corresponding barrier pair for this one should be before
- * css_rstat_updated() by the user.
+ * __css_rstat_updated() by the user.
*
* For now, there aren't any such user, so not adding the
* barrier here but if such a use-case arise, please add
@@ -614,7 +624,7 @@ static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp,
unsigned long flags)
{
u64_stats_update_end_irqrestore(&rstatbc->bsync, flags);
- css_rstat_updated(&cgrp->self, smp_processor_id());
+ __css_rstat_updated(&cgrp->self, smp_processor_id());
put_cpu_ptr(rstatbc);
}
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c03d4787d466..749c128b4fad 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -679,7 +679,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, long val,
if (!val)
return;
- css_rstat_updated(&memcg->css, cpu);
+ __css_rstat_updated(&memcg->css, cpu);
statc_pcpu = memcg->vmstats_percpu;
for (; statc_pcpu; statc_pcpu = statc->parent_pcpu) {
statc = this_cpu_ptr(statc_pcpu);
@@ -2796,7 +2796,7 @@ static inline void account_slab_nmi_safe(struct mem_cgroup *memcg,
struct mem_cgroup_per_node *pn = memcg->nodeinfo[pgdat->node_id];
/* preemption is disabled in_nmi(). */
- css_rstat_updated(&memcg->css, smp_processor_id());
+ __css_rstat_updated(&memcg->css, smp_processor_id());
if (idx == NR_SLAB_RECLAIMABLE_B)
atomic_add(nr, &pn->slab_reclaimable);
else
@@ -3019,7 +3019,7 @@ static inline void account_kmem_nmi_safe(struct mem_cgroup *memcg, int val)
mod_memcg_state(memcg, MEMCG_KMEM, val);
} else {
/* preemption is disabled in_nmi(). */
- css_rstat_updated(&memcg->css, smp_processor_id());
+ __css_rstat_updated(&memcg->css, smp_processor_id());
atomic_add(val, &memcg->kmem_stat);
}
}
--
2.53.0
^ permalink raw reply related
* [syzbot ci] Re: memcg: cache obj_stock by memcg, not by objcg pointer
From: syzbot ci @ 2026-05-16 6:58 UTC (permalink / raw)
To: akpm, cgroups, hannes, kernel-team, linux-kernel, linux-mm,
mhocko, muchun.song, oliver.sang, qi.zheng, roman.gushchin,
shakeel.butt
Cc: syzbot, syzkaller-bugs
In-Reply-To: <20260515171953.2224503-1-shakeel.butt@linux.dev>
syzbot ci has tested the following series
[v1] memcg: cache obj_stock by memcg, not by objcg pointer
https://lore.kernel.org/all/20260515171953.2224503-1-shakeel.butt@linux.dev
* [PATCH] memcg: cache obj_stock by memcg, not by objcg pointer
and found the following issue:
WARNING in __refill_obj_stock
Full report is available here:
https://ci.syzbot.org/series/8efc6e46-4b2e-43ab-90a0-62552bdc14a6
***
WARNING in __refill_obj_stock
tree: mm-new
URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/akpm/mm.git
base: 0cec77cfd5314c0b3b03530abe1a4b32e991f639
arch: amd64
compiler: Debian clang version 21.1.8 (++20251221033036+2078da43e25a-1~exp1~20251221153213.50), Debian LLD 21.1.8
config: https://ci.syzbot.org/builds/fda0a69d-af56-4b9f-8b30-b63ea4756923/config
------------[ cut here ]------------
debug_locks && !(rcu_read_lock_held() || lock_is_held(&(&cgroup_mutex)->dep_map))
WARNING: ./include/linux/memcontrol.h:380 at __refill_obj_stock+0x4fd/0x610, CPU#0: syz.1.48/5712
Modules linked in:
CPU: 0 UID: 0 PID: 5712 Comm: syz.1.48 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:__refill_obj_stock+0x4fd/0x610
Code: 89 e7 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d e9 d8 ba 00 00 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc cc 90 <0f> 0b 90 e9 a8 fb ff ff 44 89 f9 80 e1 07 80 c1 03 38 c1 0f 8c 60
RSP: 0018:ffffc9000483f4b0 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff88810beb0f80 RCX: 0000000080000001
RDX: 0000000000000110 RSI: ffffffff8e21e93e RDI: ffffffff8c28b8e0
RBP: 0000000000000001 R08: ffffffff8239a83c R09: ffff88812103c600
R10: dffffc0000000000 R11: ffffed102d89bae9 R12: 1ffff110242078c8
R13: ffff88810beb0d80 R14: dffffc0000000000 R15: ffff88812103c600
FS: 0000000000000000(0000) GS:ffff88818dc89000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f80ad948060 CR3: 000000000e74a000 CR4: 00000000000006f0
Call Trace:
<TASK>
__memcg_slab_free_hook+0x2ed/0x4b0
kmem_cache_free+0x381/0x650
__put_anon_vma+0x12b/0x2d0
unlink_anon_vmas+0x58b/0x730
free_pgtables+0x802/0xb40
exit_mmap+0x490/0x9e0
__mmput+0x118/0x430
exit_mm+0x18e/0x250
do_exit+0x6a2/0x22c0
do_group_exit+0x21b/0x2d0
get_signal+0x1284/0x1330
arch_do_signal_or_restart+0xbc/0x840
exit_to_user_mode_loop+0x8c/0x4d0
do_syscall_64+0x33e/0xf80
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f6eea99ce59
Code: Unable to access opcode bytes at 0x7f6eea99ce2f.
RSP: 002b:00007f6eeb8240e8 EFLAGS: 00000246
ORIG_RAX: 00000000000000ca
RAX: 0000000000000001 RBX: 00007f6eeac15fa8 RCX: 00007f6eea99ce59
RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007f6eeac15fac
RBP: 00007f6eeac15fa0 R08: 3fffffffffffffff R09: 0000000000000000
R10: 0000200001000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f6eeac16038 R14: 00007fffdc60cea0 R15: 00007fffdc60cf88
</TASK>
***
If these findings have caused you to resend the series or submit a
separate fix, please add the following tag to your commit message:
Tested-by: syzbot@syzkaller.appspotmail.com
---
This report is generated by a bot. It may contain errors.
syzbot ci engineers can be reached at syzkaller@googlegroups.com.
To test a patch for this bug, please reply with `#syz test`
(should be on a separate line).
The patch should be attached to the email.
Note: arguments like custom git repos and branches are not supported.
^ permalink raw reply
* Re: [PATCH] cpuset: Fix multi-source deadline task accounting and bandwidth bypass
From: Aaron Tomlin @ 2026-05-16 5:47 UTC (permalink / raw)
To: Waiman Long
Cc: Dietmar Eggemann, tj, hannes, mkoutny, chenridong, neelx, cgroups,
linux-kernel
In-Reply-To: <bda91fbe-e14f-45b1-8c61-27f16122bcc1@redhat.com>
On Thu, May 14, 2026 at 12:26:35AM -0400, Waiman Long wrote:
> > A concurrent sched_setscheduler() could alter the scheduling class of a
> > task between the initial pass and a rollback. This assertion seems valid to
> > me. Currently, neither cgroup_mutex or cpuset_mutex prevents scheduling
> > class changes.
> >
> > Should I let you handle this too?
>
> No, you can handle it if you want. I am more familiar with the cpuset code,
> but scheduler is much more complex. I don't think I have enough
> understanding of the code to handle it correctly.
Hi Longman,
I'll give it a try and rebase against your final changes.
Kind regards,
--
Aaron Tomlin
^ permalink raw reply
* Re: [PATCH] cgroup/rstat: validate cpu before css_rstat_cpu() access
From: Ming Qing @ 2026-05-16 5:25 UTC (permalink / raw)
To: Tejun Heo; +Cc: Johannes Weiner, Michal Koutný, cgroups, bpf, linux-kernel
In-Reply-To: <agdJWXRZF_pRgggt@slm.duckdns.org>
On Sat, May 16, 2026 at 12:27 AM +0800, Tejun Heo wrote:
> Can you please add this validation to the BPF kfunc that's exposing it?
Sure, will do. I'll split out an internal __css_rstat_updated() helper
and keep the cpu validation in the css_rstat_updated() kfunc wrapper.
Internal callers will be converted to the helper in v2.
^ permalink raw reply
* [PATCH] cgroup/rdma: drop unnecessary READ_ONCE() on event counters
From: Tao Cui @ 2026-05-16 5:25 UTC (permalink / raw)
To: tj, hannes, mkoutny, cgroups; +Cc: Tao Cui
All accesses to the event counters are serialized by rdmacg_mutex,
making the READ_ONCE() annotations unnecessary. Remove them.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
kernel/cgroup/rdma.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/kernel/cgroup/rdma.c b/kernel/cgroup/rdma.c
index 7c238a9d64d4..5e82a03b3270 100644
--- a/kernel/cgroup/rdma.c
+++ b/kernel/cgroup/rdma.c
@@ -145,10 +145,10 @@ static bool rpool_has_persistent_state(struct rdmacg_resource_pool *rpool)
*/
for (i = 0; i < RDMACG_RESOURCE_MAX; i++) {
if (rpool->resources[i].peak ||
- READ_ONCE(rpool->events_max[i]) ||
- READ_ONCE(rpool->events_local_max[i]) ||
- READ_ONCE(rpool->events_alloc_fail[i]) ||
- READ_ONCE(rpool->events_local_alloc_fail[i]))
+ rpool->events_max[i] ||
+ rpool->events_local_max[i] ||
+ rpool->events_alloc_fail[i] ||
+ rpool->events_local_alloc_fail[i])
return true;
}
return false;
@@ -654,9 +654,9 @@ static int rdmacg_events_show(struct seq_file *sf, void *v)
for (i = 0; i < RDMACG_RESOURCE_MAX; i++) {
seq_printf(sf, "%s.max=%llu %s.alloc_fail=%llu",
rdmacg_resource_names[i],
- rpool ? READ_ONCE(rpool->events_max[i]) : 0ULL,
+ rpool ? rpool->events_max[i] : 0ULL,
rdmacg_resource_names[i],
- rpool ? READ_ONCE(rpool->events_alloc_fail[i]) : 0ULL);
+ rpool ? rpool->events_alloc_fail[i] : 0ULL);
if (i < RDMACG_RESOURCE_MAX - 1)
seq_putc(sf, ' ');
}
@@ -683,9 +683,9 @@ static int rdmacg_events_local_show(struct seq_file *sf, void *v)
for (i = 0; i < RDMACG_RESOURCE_MAX; i++) {
seq_printf(sf, "%s.max=%llu %s.alloc_fail=%llu",
rdmacg_resource_names[i],
- rpool ? READ_ONCE(rpool->events_local_max[i]) : 0ULL,
+ rpool ? rpool->events_local_max[i] : 0ULL,
rdmacg_resource_names[i],
- rpool ? READ_ONCE(rpool->events_local_alloc_fail[i]) : 0ULL);
+ rpool ? rpool->events_local_alloc_fail[i] : 0ULL);
if (i < RDMACG_RESOURCE_MAX - 1)
seq_putc(sf, ' ');
}
--
2.43.0
^ permalink raw reply related
* Re: [PATCH cgroup/for-next v2 0/5] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Waiman Long @ 2026-05-16 4:36 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin
In-Reply-To: <20260516042448.698216-1-longman@redhat.com>
On 5/16/26 12:24 AM, Waiman Long wrote:
> Sashiko AI review of another cpuset patch had found that cpuset_attach()
> and cpuset_can_attach() can be passed a cgroup_taskset with tasks
> migrating from one source cpuset to multiple destination cpusets and
> vice versa. Further testing of the cpuset code indicates that this is
> indeed the case when the v2 cpuset controller is enabled or disabled.
>
> Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that
> there will be one source and one destinaton cpuset which may result in
> inocrrect behavior.
>
> This patch series is created to fix this issue. The first 2 patches are
> just preparatory patches to make the remaining patches easier to review.
>
> Patch 3 adds a new attach_old_cs field into task_struct to track the
> old cpuset to be used in case when cpuset_migrate_mm() needs to be
> called in cpuset_attach().
>
> Patch 4 moves mpol_rebind_mm() and cpuset_migrate_mm() inside
> cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works
> more like moving task from one cpuset to another one, while also make
> supporting multiple source and destination cpusets easier.
>
> Patch 5 makes the necessary changes to enable the support of multiple
> source and destination cpusets by keeping all the source and destination
> cpusets found during task iterations in two singly linked lists for
> source and destination cpusets respectively.
Sorry, I forgot to add the change log. It is basically to address all
the AI review comments [1] for my v1 patch.
The 2 major changes are to move cpuset_migrate_mm() into
cpuset_attach_task() and add a new field in task_struct to record the
old cpuset to be used by cpuset_migrate_mm(). There also some other
minor changes.
[1]
https://lore.kernel.org/lkml/4f49602d35d987e029b8e92a577f0c60@kernel.org/
Cheers,
Longman
>
> Waiman Long (5):
> cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
> cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
> cgroup/cpuset: Replace cpuset_attach_old_cs by a new attach_old_cs
> field in task_struct
> cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
> cpuset_attach_task()
> cgroup/cpuset: Support multiple source/destination cpusets for
> cpuset_*attach()
>
> include/linux/sched.h | 3 +
> kernel/cgroup/cpuset-internal.h | 6 +
> kernel/cgroup/cpuset.c | 358 +++++++++++++++++++++-----------
> 3 files changed, 249 insertions(+), 118 deletions(-)
>
^ permalink raw reply
* [PATCH cgroup/for-next v2 4/5] cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()
From: Waiman Long @ 2026-05-16 4:24 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin, Waiman Long
In-Reply-To: <20260516042448.698216-1-longman@redhat.com>
The cpuset_attach_task() was introduced in commit 42a11bf5c543
("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly")
to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like
moving a task from one cpuset into another one. That commits didn't
move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader
into cpuset_attach_task().
When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the
new task is its own group leader. So it is still not equivalent to
moving task between cpusets in this case. Make CLONE_INTO_CGROUP
behaves more close to cpuset_attach() by moving the mpol_rebind_mm()
and cpuset_migrate_mm() calls inside cpuset_attach_task().
Besides, the original code use cpuset_attach_nodemask_to for
both nodemask returned by guarantee_online_mems() used only by
cpuset_change_task_nodemask() and cs->effective_mems in all other cases.
Such dual use is now impractical by merging the two task iteration loops
into one. So keep cpuset_attach_nodemask_to for the nodemask returned
by guarantee_online_mems() and reference cs->effective_mems directly
in all the other cases.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 82 ++++++++++++++++++++++--------------------
1 file changed, 43 insertions(+), 39 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index fc632370d07c..ab9fcc001f79 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3130,9 +3130,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
*/
static cpumask_var_t cpus_attach;
static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;
static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
{
+ struct mm_struct *mm;
+
lockdep_assert_cpuset_lock_held();
if (cs != &top_cpuset)
@@ -3146,17 +3149,48 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
*/
WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+ if (cpuset_v2() && !attach_mems_updated)
+ return;
+
cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to);
cpuset1_update_task_spread_flags(cs, task);
+
+ if (task != task->group_leader)
+ return;
+
+ /*
+ * Change mm for threadgroup leader. This is expensive and may
+ * sleep and should be moved outside migration path proper.
+ */
+ mm = get_task_mm(task);
+ if (mm) {
+ struct cpuset *oldcs = task->attach_old_cs;
+
+ mpol_rebind_mm(mm, &cs->effective_mems);
+
+ /*
+ * old_mems_allowed is the same with mems_allowed
+ * here, except if this task is being moved
+ * automatically due to hotplug. In that case
+ * @mems_allowed has been updated and is empty, so
+ * @old_mems_allowed is the right nodesets that we
+ * migrate mm from.
+ */
+ if (oldcs && is_memory_migrate(cs)) {
+ cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
+ &cs->effective_mems);
+ queue_task_work = true;
+ } else {
+ mmput(mm);
+ }
+ }
}
static void cpuset_attach(struct cgroup_taskset *tset)
{
struct task_struct *task;
- struct task_struct *leader;
struct cgroup_subsys_state *css;
struct cpuset *cs, *oldcs;
- bool queue_task_work = false;
task = cgroup_taskset_first(tset, &css);
oldcs = task->attach_old_cs;
@@ -3164,6 +3198,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */
mutex_lock(&cpuset_mutex);
+ queue_task_work = false;
/*
* In the default hierarchy, enabling cpuset in the child cgroups
@@ -3171,53 +3206,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)
* in effective cpus and mems. In that case, we can optimize out
* by skipping the task iteration and update.
*/
- if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated) {
- cpuset_attach_nodemask_to = cs->effective_mems;
+ if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated)
goto out;
- }
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
cgroup_taskset_for_each(task, css, tset)
cpuset_attach_task(cs, task);
- /*
- * Change mm for all threadgroup leaders. This is expensive and may
- * sleep and should be moved outside migration path proper. Skip it
- * if there is no change in effective_mems and CS_MEMORY_MIGRATE is
- * not set.
- */
- cpuset_attach_nodemask_to = cs->effective_mems;
- if (!is_memory_migrate(cs) && !attach_mems_updated)
- goto out;
-
- cgroup_taskset_for_each_leader(leader, css, tset) {
- struct mm_struct *mm = get_task_mm(leader);
-
- if (mm) {
- mpol_rebind_mm(mm, &cpuset_attach_nodemask_to);
-
- /*
- * old_mems_allowed is the same with mems_allowed
- * here, except if this task is being moved
- * automatically due to hotplug. In that case
- * @mems_allowed has been updated and is empty, so
- * @old_mems_allowed is the right nodesets that we
- * migrate mm from.
- */
- if (is_memory_migrate(cs)) {
- cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
- &cpuset_attach_nodemask_to);
- queue_task_work = true;
- } else
- mmput(mm);
- }
- }
-
out:
if (queue_task_work)
schedule_flush_migrate_mm();
- cs->old_mems_allowed = cpuset_attach_nodemask_to;
+ cs->old_mems_allowed = cs->effective_mems;
if (cs->nr_migrate_dl_tasks) {
cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
@@ -3667,7 +3667,11 @@ static void cpuset_fork(struct task_struct *task)
/* CLONE_INTO_CGROUP */
mutex_lock(&cpuset_mutex);
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+ /* Assume CPUs and memory nodes are updated */
+ attach_cpus_updated = attach_mems_updated = true;
+ task->attach_old_cs = task_cs(current);
cpuset_attach_task(cs, task);
+ attach_cpus_updated = attach_mems_updated = false;
dec_attach_in_progress_locked(cs);
mutex_unlock(&cpuset_mutex);
--
2.54.0
^ permalink raw reply related
* [PATCH cgroup/for-next v2 5/5] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Waiman Long @ 2026-05-16 4:24 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin, Waiman Long
In-Reply-To: <20260516042448.698216-1-longman@redhat.com>
With cgroup v2, the cgroup_taskset structure passed into the cgroup
can_attach() and attach() methods can contain task migration data with
multiple destination or source cpusets when the cpuset controller is
enabled or disabled respectively.
Since cpuset is threaded, another possible way to cause many-to-one
migration is to move the whole process with multiple threads in different
cpuset enabled threaded cgroups into another cpuset enabled cgroup.
Alternatively, multiple processs from different cpusets can be written
into cgroup.proc as a single operation.
The current cpuset_can_attach() and cpuset_attach() functions still
expect task migration is from one source cpuset to one destination
cpuset. This has been the case since cpuset was enabled for cgroup v2
in commit 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default
hierarchy").
This problem is less an issue when enabling the cpuset controller as all
the newly created child cpusets will have exactly the same set of CPUs
and memory nodes except when deadline tasks are involved in migration
as the deadline task accounting data can be off.
It can be more problematic when the cpuset controller is disabled as
their set of CPUs and memory nodes may differ from their parent or with
the moving of multi-threaded process from different threaded cgroups.
Fix that by tracking the set of source (old) and destination cpusets
in singly linked lists and iterating them all to properly update the
internal data. Also keep the current cs and oldcs variables up-to-date
with the css and task iterators. cpuset_attach_old_cs is now dropped
as the old cpusets are now being tracked.
To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both
the source and destination cpusets are decremented/incremented with
their values added to nr_deadline_tasks when the migration is successful.
Fixes: 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy")
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset-internal.h | 6 +
kernel/cgroup/cpuset.c | 212 +++++++++++++++++++++++---------
2 files changed, 161 insertions(+), 57 deletions(-)
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index f7aaf01f7cd5..4c2772a7fd5e 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -161,6 +161,12 @@ struct cpuset {
*/
bool remote_partition;
+ /*
+ * cpuset_can_attach() and cpuset_attach() specific data
+ */
+ bool attach_node_in_llist;
+ struct llist_node attach_node;
+
/*
* number of SCHED_DEADLINE tasks attached to this cpuset, so that we
* know when to rebuild associated root domain bandwidth information.
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index ab9fcc001f79..1b6eb8cf0bcd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -37,6 +37,7 @@
#include <linux/wait.h>
#include <linux/workqueue.h>
#include <linux/task_work.h>
+#include <linux/llist.h>
DEFINE_STATIC_KEY_FALSE(cpusets_pre_enable_key);
DEFINE_STATIC_KEY_FALSE(cpusets_enabled_key);
@@ -2968,6 +2969,8 @@ static int update_prstate(struct cpuset *cs, int new_prs)
* cpuset_can_attach() and cpuset_attach() specific internal data
* Protected by cpuset_mutex
*/
+static LLIST_HEAD(src_cs_head);
+static LLIST_HEAD(dst_cs_head);
static bool attach_cpus_updated;
static bool attach_mems_updated;
@@ -2980,9 +2983,10 @@ static bool attach_mems_updated;
* Also set the boolean flag passed in by @psetsched depending on if
* security_task_setscheduler() call is needed and @oldcs is not NULL.
*/
-static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
- bool *psetsched)
+static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs, bool *psetsched)
{
+ bool cpu_match, mem_match;
+
if (cpumask_empty(cs->effective_cpus) ||
(!is_in_v2_mode() && nodes_empty(cs->mems_allowed)))
return -ENOSPC;
@@ -2993,15 +2997,34 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
/*
* Update attach specific data
*/
- attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
- attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+ if (!cs->attach_node_in_llist) {
+ llist_add(&cs->attach_node, &dst_cs_head);
+ cs->attach_node_in_llist = true;
+ }
+ if (!oldcs->attach_node_in_llist) {
+ llist_add(&oldcs->attach_node, &src_cs_head);
+ oldcs->attach_node_in_llist = true;
+ }
+
+ cpu_match = cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+ mem_match = nodes_equal(cs->effective_mems, oldcs->effective_mems);
+
+ /*
+ * Set the updated flags whenever there is a mismatch in any of the
+ * src/dst pairs.
+ */
+ if (!attach_cpus_updated)
+ attach_cpus_updated = !cpu_match;
+
+ if (!attach_mems_updated)
+ attach_mems_updated = !mem_match;
/*
* Skip rights over task setsched check in v2 when nothing changes,
* migration permission derives from hierarchy ownership in
* cgroup_procs_write_permission()).
*/
- *psetsched = !cpuset_v2() || attach_cpus_updated || attach_mems_updated;
+ *psetsched = !cpuset_v2() || !cpu_match || !mem_match;
/*
* A v1 cpuset with tasks will have no CPU left only when CPU hotplug
@@ -3016,33 +3039,105 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
return 0;
}
-static int cpuset_reserve_dl_bw(struct cpuset *cs)
+/*
+ * If reset_dl_bw is set, reset the previous dl_bw_alloc() call. Otherwise,
+ * update nr_deadline_tasks according to nr_migrate_dl_tasks in both source
+ * and destination cpusets.
+ */
+static void clear_attach_data(bool reset_dl_bw)
{
+ struct cpuset *cs, *next;
+
+ llist_for_each_entry_safe(cs, next, src_cs_head.first, attach_node) {
+ cs->attach_node.next = NULL;
+ cs->attach_node_in_llist = false;
+ if (cs->nr_migrate_dl_tasks && !reset_dl_bw)
+ cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
+ cs->nr_migrate_dl_tasks = 0;
+ }
+
+ llist_for_each_entry_safe(cs, next, dst_cs_head.first, attach_node) {
+ cs->attach_node.next = NULL;
+ cs->attach_node_in_llist = false;
+ if (reset_dl_bw && cs->dl_bw_cpu >= 0)
+ dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
+ if (cs->nr_migrate_dl_tasks && !reset_dl_bw)
+ cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
+ cs->nr_migrate_dl_tasks = 0;
+ cs->sum_migrate_dl_bw = 0;
+ cs->dl_bw_cpu = -1;
+ }
+
+ src_cs_head.first = NULL;
+ dst_cs_head.first = NULL;
+ attach_cpus_updated = false;
+ attach_mems_updated = false;
+}
+
+static int cpuset_reserve_dl_bw(void)
+{
+ struct cpuset *cs;
int cpu, ret;
- if (!cs->sum_migrate_dl_bw)
- return 0;
+ llist_for_each_entry(cs, dst_cs_head.first, attach_node) {
+ if (!cs->sum_migrate_dl_bw)
+ continue;
- cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
- if (unlikely(cpu >= nr_cpu_ids))
- return -EINVAL;
+ cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
+ if (unlikely(cpu >= nr_cpu_ids))
+ return -EINVAL;
- ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
- if (ret)
- return ret;
+ ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+ if (ret)
+ return ret;
- cs->dl_bw_cpu = cpu;
+ cs->dl_bw_cpu = cpu;
+ }
return 0;
}
-static void reset_migrate_dl_data(struct cpuset *cs)
+static void set_attach_in_progress(void)
+{
+ struct cpuset *cs;
+
+ /*
+ * Mark attach is in progress. This makes validate_change() fail
+ * changes which zero cpus/mems_allowed.
+ */
+ llist_for_each_entry(cs, dst_cs_head.first, attach_node)
+ cs->attach_in_progress++;
+}
+
+static void reset_attach_in_progress(void)
{
- cs->nr_migrate_dl_tasks = 0;
- cs->sum_migrate_dl_bw = 0;
- cs->dl_bw_cpu = -1;
+ struct cpuset *cs;
+
+ llist_for_each_entry(cs, dst_cs_head.first, attach_node)
+ dec_attach_in_progress_locked(cs);
}
-/* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */
+/*
+ * Called by cgroups to determine if a cpuset is usable; cpuset_mutex held.
+ *
+ * With cgroup v2, enabling of cpuset controller in a cgroup subtree can
+ * cause @tset to contain task migration data from one parent cpuset to multiple
+ * child cpusets. Not much is needed to be done here other than tracking the
+ * number of DL tasks in each cpuset as the CPUs and memory nodes of the child
+ * cpusets are exactly the same as the parent.
+ *
+ * Conversely, disabling of cpuset controller can cause @tset to contain task
+ * migration data from multiple child cpusets to one parent cpuset. Here, the
+ * CPUs and memory nodes of the child cpusets may be different from the parent,
+ * but must be a subset of its parent.
+ *
+ * Another possible many-to-one migration is the moving of the whole
+ * multithreaded process with threads in different cpusets to another cpuset.
+ * Alternatively, multiple processes from multiple cpusets can be moved to
+ * another cpuset in a single operation.
+ *
+ * For all other use cases including cgroup v1, @tset task migration data
+ * should be from one source cpuset to one destination cpuset.
+ */
static int cpuset_can_attach(struct cgroup_taskset *tset)
{
struct cgroup_subsys_state *css;
@@ -3062,6 +3157,16 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
goto out_unlock;
cgroup_taskset_for_each(task, css, tset) {
+ struct cpuset *newcs = css_cs(css);
+ struct cpuset *new_oldcs = task_cs(task);
+
+ if ((newcs != cs) || (new_oldcs != oldcs)) {
+ cs = newcs;
+ oldcs = new_oldcs;
+ ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
+ if (ret)
+ goto out_unlock;
+ }
ret = task_can_attach(task);
if (ret)
goto out_unlock;
@@ -3081,23 +3186,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
* contribute to sum_migrate_dl_bw.
*/
cs->nr_migrate_dl_tasks++;
+ oldcs->nr_migrate_dl_tasks--;
if (dl_task_needs_bw_move(task, cs->effective_cpus))
cs->sum_migrate_dl_bw += task->dl.dl_bw;
}
}
- ret = cpuset_reserve_dl_bw(cs);
+ ret = cpuset_reserve_dl_bw();
out_unlock:
- if (ret) {
- reset_migrate_dl_data(cs);
- } else {
- /*
- * Mark attach is in progress. This makes validate_change() fail
- * changes which zero cpus/mems_allowed.
- */
- cs->attach_in_progress++;
- }
+ if (ret)
+ clear_attach_data(true);
+ else
+ set_attach_in_progress();
mutex_unlock(&cpuset_mutex);
return ret;
@@ -3112,14 +3213,8 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
cs = css_cs(css);
mutex_lock(&cpuset_mutex);
- dec_attach_in_progress_locked(cs);
-
- if (cs->dl_bw_cpu >= 0)
- dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
-
- if (cs->nr_migrate_dl_tasks)
- reset_migrate_dl_data(cs);
-
+ reset_attach_in_progress();
+ clear_attach_data(true);
mutex_unlock(&cpuset_mutex);
}
@@ -3190,43 +3285,46 @@ static void cpuset_attach(struct cgroup_taskset *tset)
{
struct task_struct *task;
struct cgroup_subsys_state *css;
- struct cpuset *cs, *oldcs;
+ struct cpuset *cs;
- task = cgroup_taskset_first(tset, &css);
- oldcs = task->attach_old_cs;
+ cgroup_taskset_first(tset, &css);
cs = css_cs(css);
-
lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */
mutex_lock(&cpuset_mutex);
queue_task_work = false;
/*
* In the default hierarchy, enabling cpuset in the child cgroups
- * will trigger a number of cpuset_attach() calls with no change
- * in effective cpus and mems. In that case, we can optimize out
- * by skipping the task iteration and update.
+ * will trigger a cpuset_attach() call with no change in effective cpus
+ * and mems. In that case, we can optimize out by skipping the task
+ * iteration and update, but the destination cpuset list is iterated to
+ * set old_mems_sllowed.
*/
- if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated)
+ if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated) {
+ llist_for_each_entry(cs, dst_cs_head.first, attach_node)
+ cs->old_mems_allowed = cs->effective_mems;
goto out;
+ }
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
- cgroup_taskset_for_each(task, css, tset)
+ cgroup_taskset_for_each(task, css, tset) {
+ struct cpuset *newcs = css_cs(css);
+
+ if (newcs != cs) {
+ cs->old_mems_allowed = cs->effective_mems;
+ cs = newcs;
+ guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+ }
cpuset_attach_task(cs, task);
+ }
-out:
if (queue_task_work)
schedule_flush_migrate_mm();
cs->old_mems_allowed = cs->effective_mems;
-
- if (cs->nr_migrate_dl_tasks) {
- cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
- oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks;
- reset_migrate_dl_data(cs);
- }
-
- dec_attach_in_progress_locked(cs);
-
+out:
+ reset_attach_in_progress();
+ clear_attach_data(false);
mutex_unlock(&cpuset_mutex);
}
--
2.54.0
^ permalink raw reply related
* [PATCH cgroup/for-next v2 3/5] cgroup/cpuset: Replace cpuset_attach_old_cs by a new attach_old_cs field in task_struct
From: Waiman Long @ 2026-05-16 4:24 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin, Waiman Long
In-Reply-To: <20260516042448.698216-1-longman@redhat.com>
In cpuset_can_attach(), the source (old) cpuset of the tasks is
stored in an internal cpuset_attach_old_cs variable to be used later
in cpuset_attach(). It is because such task to old cpuset information
is no longer available when cpuset_attach() is called and it is assumed
that there is only one source cpuset.
To support cgroup_taskset containing tasks from multiple source
cpusets, such an approach will no longer work. The easier way to get
the old cpuset information is to temporarily store that information
in the task_struct itself at cpuset_can_attach() and reuse it in
cpuset_attach(). However, that does increase the size of task_struct
by a 8 bytes for 64-bit kernel.
Add a new attach_old_cs field into task_struct for such purpose and
retire the cpuset_attach_old_cs internal variable.
Even though attach_old_cs can be counted as a reference to the
old cpuset, like cpuset_attach_old_cs, it is strictly used only
for communication between cpuset_can_attach() and cpuset_attach()
within the same task migration session where cgroup_mutex will be held
throughout. So no actual reference counting will be performed.
Signed-off-by: Waiman Long <longman@redhat.com>
---
include/linux/sched.h | 3 +++
kernel/cgroup/cpuset.c | 13 ++++++-------
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 004e6d56a499..9b6bb1603592 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -63,6 +63,7 @@ struct bpf_run_ctx;
struct bpf_net_context;
struct capture_control;
struct cfs_rq;
+struct cpuset;
struct fs_struct;
struct futex_pi_state;
struct io_context;
@@ -1317,6 +1318,8 @@ struct task_struct {
/* Sequence number to catch updates: */
seqcount_spinlock_t mems_allowed_seq;
int cpuset_mem_spread_rotor;
+ /* Old cpuset to be used in cpuset_attach() */
+ struct cpuset *attach_old_cs;
#endif
#ifdef CONFIG_CGROUPS
/* Control Group info protected by css_set_lock: */
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 0d01b66f464d..fc632370d07c 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2968,7 +2968,6 @@ static int update_prstate(struct cpuset *cs, int new_prs)
* cpuset_can_attach() and cpuset_attach() specific internal data
* Protected by cpuset_mutex
*/
-static struct cpuset *cpuset_attach_old_cs;
static bool attach_cpus_updated;
static bool attach_mems_updated;
@@ -3052,9 +3051,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
bool setsched_check;
int ret;
- /* used later by cpuset_attach() */
- cpuset_attach_old_cs = task_cs(cgroup_taskset_first(tset, &css));
- oldcs = cpuset_attach_old_cs;
+ oldcs = task_cs(cgroup_taskset_first(tset, &css));
cs = css_cs(css);
mutex_lock(&cpuset_mutex);
@@ -3075,6 +3072,8 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
goto out_unlock;
}
+ /* Save a copy of oldcs to be used later in cpuset_attach() */
+ task->attach_old_cs = oldcs;
if (dl_task(task)) {
/*
* Count all migrating DL tasks for cpuset task accounting.
@@ -3156,11 +3155,11 @@ static void cpuset_attach(struct cgroup_taskset *tset)
struct task_struct *task;
struct task_struct *leader;
struct cgroup_subsys_state *css;
- struct cpuset *cs;
- struct cpuset *oldcs = cpuset_attach_old_cs;
+ struct cpuset *cs, *oldcs;
bool queue_task_work = false;
- cgroup_taskset_first(tset, &css);
+ task = cgroup_taskset_first(tset, &css);
+ oldcs = task->attach_old_cs;
cs = css_cs(css);
lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */
--
2.54.0
^ permalink raw reply related
* [PATCH cgroup/for-next v2 2/5] cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
From: Waiman Long @ 2026-05-16 4:24 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin, Waiman Long
In-Reply-To: <20260516042448.698216-1-longman@redhat.com>
Expand the scope of cpuset_can_attach_check() by including the setting
of setsched flag inside cpuset_can_attach_check() with the new @oldcs
and @psetsched argument. As cpuset_can_attach_check() is also called
from cpuset_can_fork(), set the new arguments to NULL from that caller.
While at it, expose the source and destination cpuset cpu/memory check
results in the new attach_cpus_updated and attach_mems_updated static
flags so that these flags can be used directly from cpuset_attach()
without the need to do the same computations again.
No functional change is expected.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 70 +++++++++++++++++++++++++-----------------
1 file changed, 42 insertions(+), 28 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 7cae47829013..0d01b66f464d 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2964,19 +2964,56 @@ static int update_prstate(struct cpuset *cs, int new_prs)
return 0;
}
+/*
+ * cpuset_can_attach() and cpuset_attach() specific internal data
+ * Protected by cpuset_mutex
+ */
static struct cpuset *cpuset_attach_old_cs;
+static bool attach_cpus_updated;
+static bool attach_mems_updated;
/*
* Check to see if a cpuset can accept a new task
* For v1, cpus_allowed and mems_allowed can't be empty.
* For v2, effective_cpus can't be empty.
* Note that in v1, effective_cpus = cpus_allowed.
+ *
+ * Also set the boolean flag passed in by @psetsched depending on if
+ * security_task_setscheduler() call is needed and @oldcs is not NULL.
*/
-static int cpuset_can_attach_check(struct cpuset *cs)
+static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
+ bool *psetsched)
{
if (cpumask_empty(cs->effective_cpus) ||
(!is_in_v2_mode() && nodes_empty(cs->mems_allowed)))
return -ENOSPC;
+
+ if (!oldcs)
+ return 0;
+
+ /*
+ * Update attach specific data
+ */
+ attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+ attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+
+ /*
+ * Skip rights over task setsched check in v2 when nothing changes,
+ * migration permission derives from hierarchy ownership in
+ * cgroup_procs_write_permission()).
+ */
+ *psetsched = !cpuset_v2() || attach_cpus_updated || attach_mems_updated;
+
+ /*
+ * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
+ * brings the last online CPU offline as users are not allowed to empty
+ * cpuset.cpus when there are active tasks inside. When that happens,
+ * we should allow tasks to migrate out without security check to make
+ * sure they will be able to run after migration.
+ */
+ if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
+ *psetsched = false;
+
return 0;
}
@@ -3023,29 +3060,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
mutex_lock(&cpuset_mutex);
/* Check to see if task is allowed in the cpuset */
- ret = cpuset_can_attach_check(cs);
+ ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
if (ret)
goto out_unlock;
- /*
- * Skip rights over task setsched check in v2 when nothing changes,
- * migration permission derives from hierarchy ownership in
- * cgroup_procs_write_permission()).
- */
- setsched_check = !cpuset_v2() ||
- !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus) ||
- !nodes_equal(cs->effective_mems, oldcs->effective_mems);
-
- /*
- * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
- * brings the last online CPU offline as users are not allowed to empty
- * cpuset.cpus when there are active tasks inside. When that happens,
- * we should allow tasks to migrate out without security check to make
- * sure they will be able to run after migration.
- */
- if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
- setsched_check = false;
-
cgroup_taskset_for_each(task, css, tset) {
ret = task_can_attach(task);
if (ret)
@@ -3140,7 +3158,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
struct cgroup_subsys_state *css;
struct cpuset *cs;
struct cpuset *oldcs = cpuset_attach_old_cs;
- bool cpus_updated, mems_updated;
bool queue_task_work = false;
cgroup_taskset_first(tset, &css);
@@ -3148,9 +3165,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */
mutex_lock(&cpuset_mutex);
- cpus_updated = !cpumask_equal(cs->effective_cpus,
- oldcs->effective_cpus);
- mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
/*
* In the default hierarchy, enabling cpuset in the child cgroups
@@ -3158,7 +3172,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
* in effective cpus and mems. In that case, we can optimize out
* by skipping the task iteration and update.
*/
- if (cpuset_v2() && !cpus_updated && !mems_updated) {
+ if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated) {
cpuset_attach_nodemask_to = cs->effective_mems;
goto out;
}
@@ -3175,7 +3189,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
* not set.
*/
cpuset_attach_nodemask_to = cs->effective_mems;
- if (!is_memory_migrate(cs) && !mems_updated)
+ if (!is_memory_migrate(cs) && !attach_mems_updated)
goto out;
cgroup_taskset_for_each_leader(leader, css, tset) {
@@ -3590,7 +3604,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
mutex_lock(&cpuset_mutex);
/* Check to see if task is allowed in the cpuset */
- ret = cpuset_can_attach_check(cs);
+ ret = cpuset_can_attach_check(cs, NULL, NULL);
if (ret)
goto out_unlock;
--
2.54.0
^ permalink raw reply related
* [PATCH cgroup/for-next v2 1/5] cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
From: Waiman Long @ 2026-05-16 4:24 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin, Waiman Long
In-Reply-To: <20260516042448.698216-1-longman@redhat.com>
Extract the DL bandwidth allocation code in cpuset_attach() to a new
cpuset_reserve_dl_bw() helper to simplify code.
No functional change is expected.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 53 ++++++++++++++++++++++++------------------
1 file changed, 30 insertions(+), 23 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index bcefc9f50ac5..7cae47829013 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2980,6 +2980,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)
return 0;
}
+static int cpuset_reserve_dl_bw(struct cpuset *cs)
+{
+ int cpu, ret;
+
+ if (!cs->sum_migrate_dl_bw)
+ return 0;
+
+ cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
+ if (unlikely(cpu >= nr_cpu_ids))
+ return -EINVAL;
+
+ ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+ if (ret)
+ return ret;
+
+ cs->dl_bw_cpu = cpu;
+ return 0;
+}
+
static void reset_migrate_dl_data(struct cpuset *cs)
{
cs->nr_migrate_dl_tasks = 0;
@@ -2994,7 +3013,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
struct cpuset *cs, *oldcs;
struct task_struct *task;
bool setsched_check;
- int cpu, ret;
+ int ret;
/* used later by cpuset_attach() */
cpuset_attach_old_cs = task_cs(cgroup_taskset_first(tset, &css));
@@ -3050,31 +3069,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
}
}
- if (!cs->sum_migrate_dl_bw)
- goto out_success;
-
- cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
- if (unlikely(cpu >= nr_cpu_ids)) {
- ret = -EINVAL;
- goto out_unlock;
- }
-
- ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
- if (ret)
- goto out_unlock;
-
- cs->dl_bw_cpu = cpu;
-
-out_success:
- /*
- * Mark attach is in progress. This makes validate_change() fail
- * changes which zero cpus/mems_allowed.
- */
- cs->attach_in_progress++;
+ ret = cpuset_reserve_dl_bw(cs);
out_unlock:
- if (ret)
+ if (ret) {
reset_migrate_dl_data(cs);
+ } else {
+ /*
+ * Mark attach is in progress. This makes validate_change() fail
+ * changes which zero cpus/mems_allowed.
+ */
+ cs->attach_in_progress++;
+ }
+
mutex_unlock(&cpuset_mutex);
return ret;
}
--
2.54.0
^ permalink raw reply related
* [PATCH cgroup/for-next v2 0/5] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Waiman Long @ 2026-05-16 4:24 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin, Waiman Long
Sashiko AI review of another cpuset patch had found that cpuset_attach()
and cpuset_can_attach() can be passed a cgroup_taskset with tasks
migrating from one source cpuset to multiple destination cpusets and
vice versa. Further testing of the cpuset code indicates that this is
indeed the case when the v2 cpuset controller is enabled or disabled.
Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that
there will be one source and one destinaton cpuset which may result in
inocrrect behavior.
This patch series is created to fix this issue. The first 2 patches are
just preparatory patches to make the remaining patches easier to review.
Patch 3 adds a new attach_old_cs field into task_struct to track the
old cpuset to be used in case when cpuset_migrate_mm() needs to be
called in cpuset_attach().
Patch 4 moves mpol_rebind_mm() and cpuset_migrate_mm() inside
cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works
more like moving task from one cpuset to another one, while also make
supporting multiple source and destination cpusets easier.
Patch 5 makes the necessary changes to enable the support of multiple
source and destination cpusets by keeping all the source and destination
cpusets found during task iterations in two singly linked lists for
source and destination cpusets respectively.
Waiman Long (5):
cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
cgroup/cpuset: Replace cpuset_attach_old_cs by a new attach_old_cs
field in task_struct
cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
cpuset_attach_task()
cgroup/cpuset: Support multiple source/destination cpusets for
cpuset_*attach()
include/linux/sched.h | 3 +
kernel/cgroup/cpuset-internal.h | 6 +
kernel/cgroup/cpuset.c | 358 +++++++++++++++++++++-----------
3 files changed, 249 insertions(+), 118 deletions(-)
--
2.54.0
^ permalink raw reply
* Re: [PATCH v2 00/10] sched: Flatten the pick
From: Qais Yousef @ 2026-05-16 3:30 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak
In-Reply-To: <20260511113104.563854162@infradead.org>
On 05/11/26 13:31, Peter Zijlstra wrote:
> Hi!
>
> So cgroup scheduling has always been a pain in the arse. The problems start
> with weight distribution and end with hierachical picks and it all sucks.
It does..
Not that it is useful info, but we talked briefly about it at OSPM, so thought
I'll report back. I gave this a go with my test case from schedqos announcement
[1] of running schbench with kernel build as BACKGROUND noise, but the fairness
imposed at group level is preserved (as expected) even if the pick is
flattened.
I do actually want a total flat system, ie: disable this whole thing :-)
The problem is to create a system where you want to introduce smart tagging
based on tasks, group scheduling becomes a big problem. If a task is set as
background or interactive, it has to be global to be enforced otherwise it
loses its meaning. And my test case stresses two long running tasks one is
interactive but the other is background and group scheduling imposes fairness
that breaks the task level tagging. Managing deadline via runtime doesn't help
here since they are both always busy tasks; and one must use nice values to
manage bandwidth. But nice values are local when autogroup/cgroups are present.
I need to find a simple way to turn this thing off at runtime and properly
flatten it ;-)
/runs away
[1] https://lore.kernel.org/lkml/20260415000910.2h5misvwc45bdumu@airbuntu/
^ permalink raw reply
* Re: [PATCH cgroup/for-next 0/4] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Waiman Long @ 2026-05-15 23:13 UTC (permalink / raw)
To: Tejun Heo
Cc: Chen Ridong, Johannes Weiner, Michal Koutný, cgroups,
linux-kernel, Dietmar Eggemann, Aaron Tomlin, Juri Lelli
In-Reply-To: <4f49602d35d987e029b8e92a577f0c60@kernel.org>
On 5/14/26 5:46 PM, Tejun Heo wrote:
> Hello,
>
> Quick AI-assisted review pass; passing the points along for human eyes.
Thank for passing the AI review info.
> Patch 4:
>
> - The leader loop comment says "Only v1 supports memory_migrate", but
> CS_MEMORY_MIGRATE is set unconditionally on v2 cpusets in
> cpuset_css_alloc(). With v2 controller-disable folding children with
> differing effective_mems into the parent, picking a single
> llist_entry(src_cs_head.first, ...) as oldcs passes the wrong source
> nodemask to cpuset_migrate_mm() for every leader whose actual source
> differs. Looks like the source needs to be looked up per leader.
I falsely assumes that CS_MEMORY_MIGRATE isn't set for v2. I am going to
move the cpuset_migrate_mm() call into cpuset_attach_task() as well. To
enable access to right oldcs information, I have to reluctantly add the
oldcs pointer to task_struct as that is the simplest way moving forward.
>
> - cs->old_mems_allowed updates are inconsistent across destinations: the
> mid-loop transition assigns cs->effective_mems (raw) while the tail
> assignment uses cpuset_attach_nodemask_to (after guarantee_online_mems).
> The v2 fast-path also updates only the first-task cs, leaving other
> destinations on dst_cs_head stale.
I will clarify the confusion and fix the v2 fast-path in v2.
> Patch 3:
>
> - Changelog says "the newly cloned task isn't the group leader", but for
> CLONE_INTO_CGROUP without CLONE_THREAD the new task is its own
> group_leader, so the new mpol_rebind_mm() block in cpuset_attach_task()
> does run from cpuset_fork(). Either acknowledge as an incidental
> improvement or guard the new path.
Will update the patch description.
>
> Patch 1:
>
> - alloc_dl_bw() reads confusingly next to the scheduler's dl_bw_alloc()
> while doing more (pick cpu, call dl_bw_alloc, record cs->dl_bw_cpu).
> Something like cpuset_reserve_dl_bw() would be clearer.
Sure, cpuset_reserve_dl_bw() sounds like a good name.
> - The relocated "Mark attach is in progress" comment sits inside a
> braceless else; either move it above the if (ret) or brace both arms.
Will add the additional braces though that will be removed in a later patch.
Cheers,
Longman
>
> Patch 2 looked clean.
>
> Thanks.
>
> --
> tejun
>
^ permalink raw reply
* Re: [PATCH] memcg: cache obj_stock by memcg, not by objcg pointer
From: Shakeel Butt @ 2026-05-15 18:42 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260515171953.2224503-1-shakeel.butt@linux.dev>
On Fri, May 15, 2026 at 10:19:53AM -0700, Shakeel Butt wrote:
> Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
> per-node type") split a memcg's single obj_cgroup into one per NUMA
> node, but the per-CPU obj_stock_pcp still keys cached_objcg by
> pointer. Cross-NUMA workloads now see a drain on every refill and a
> miss on every consume that targets a sibling per-node objcg of the
> same memcg, producing the 67.7% stress-ng switch-mq regression
> reported by LKP.
>
> stock->nr_bytes are fungible across per-node objcgs of one memcg:
> drain_obj_stock() and obj_cgroup_uncharge_pages() both account via
> obj_cgroup_memcg(). Treat the cache as keyed by memcg in both
> __consume_obj_stock() and __refill_obj_stock() so siblings share the
> reserve -- eliminating the drain on free and keeping the alloc fast
> path in consume.
>
> Though kernel test robot reported the regression but it was not easy to
> reproduce locally. Qi implemented [1] a specialized reproducer to show
> the corner case which cause the regression and then Qi tested the patch
> and reported that the corner case is eliminated after the patch.
>
> Reported-by: kernel test robot <oliver.sang@intel.com>
> Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
> Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> Link: https://lore.kernel.org/19693be6-7132-446e-b3fc-b7e9f56e5949@linux.dev/ [1]
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Debugged-by: Qi Zheng <qi.zheng@linux.dev>
> Tested-by: Qi Zheng <qi.zheng@linux.dev>
Pasting the sashiko review here as it seems interesting and good:
May 15, 2026 at 10:52 AM, sashiko-bot@kernel.org mailto:sashiko-bot@kernel.org wrote:
>
> Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
> - [High] Fungible sharing of stock reserve bytes between per-node objcgs breaks the obj_cgroup_release invariant, causing permanent kmem charge leaks and WARN_ON_ONCE.
> - [High] Missing update to `__account_obj_stock()` bypasses vmstat batching for sibling objcgs, causing severe lock and atomic contention.
> - [High] Calling `obj_cgroup_memcg()` without required RCU or cgroup locks triggers lockdep splats on fast paths.
> --
>
> commit e77fe5b6af93b69e06c1ccd13f83e26b59234483
> Author: Shakeel Butt <shakeel.butt@linux.dev>
>
> memcg: cache obj_stock by memcg, not by objcg pointer
>
> This commit modifies the per-CPU object stock caching to share the reserve
> among per-node sibling obj_cgroups of the same memcg, aiming to reduce
> drains and refills on cross-NUMA workloads.
>
> Link: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
> Link: https://lore.kernel.org/19693be6-7132-446e-b3fc-b7e9f56e5949@linux.dev/
>
> >
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index d978e18b9b2d0..66448f4285316 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -3210,7 +3210,11 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg,
> > struct obj_stock_pcp *stock,
> > unsigned int nr_bytes)
> > {
> > - if (objcg == READ_ONCE(stock->cached_objcg) &&
> > + struct obj_cgroup *cached = READ_ONCE(stock->cached_objcg);
> > +
> > + /* Cache is keyed by memcg; sibling per-node objcgs share the reserve. */
> > + if ((cached == objcg ||
> > + (cached && obj_cgroup_memcg(cached) == obj_cgroup_memcg(objcg))) &&
> >
> Does calling obj_cgroup_memcg() here trigger lockdep warnings on slab fast
> paths?
>
> Looking at obj_cgroup_memcg(), it expects rcu_read_lock_held() or
> lockdep_is_held(&cgroup_mutex) to be true.
>
> Since this function is called from slab fast paths where only the local
> CPU stock lock is held, the RCU read lock may not be held.
>
> Could this check directly compare READ_ONCE(cached->memcg) and
> READ_ONCE(objcg->memcg) to avoid the lockdep assertion?
Yes, this makes sense as we are not really dereferencing the memcg here.
>
> >
> > stock->nr_bytes >= nr_bytes) {
> > stock->nr_bytes -= nr_bytes;
> > return true;
> >
> Will sharing the reserve between per-node sibling objcgs without updating
> stock->cached_objcg break the page multiple invariant in
> obj_cgroup_release()?
>
> If an allocation for objcg_B consumes bytes originally funded by objcg_A,
> and the stock is later drained, those borrowed bytes are flushed into
> objcg_A->nr_charged_bytes.
>
> When obj_cgroup_release() is invoked, nr_charged_bytes will not be an
> exact multiple of PAGE_SIZE. Will this trigger
> WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1)) and truncate the remainder,
> permanently leaking the page charge from the memcg?
This is actually a very good point and need more thought.
>
> [ ... ]
>
> >
> > @@ -3318,6 +3322,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > unsigned int nr_bytes,
> > bool allow_uncharge)
> > {
> >
> [ ... ]
>
> >
> > - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> > + cached = READ_ONCE(stock->cached_objcg);
> > + /* Same memcg: bytes are fungible, no drain needed. */
> > + if (cached != objcg &&
> > + (!cached || obj_cgroup_memcg(cached) != obj_cgroup_memcg(objcg))) {
> > drain_obj_stock(stock);
> >
> Since the stock->cached_objcg pointer is not updated when a sibling is
> already cached, does this bypass the vmstat batching in
> __account_obj_stock()?
>
> If __account_obj_stock() still requires strict pointer equality
> (READ_ONCE(stock->cached_objcg) == objcg), alternating cross-node
> allocations will continually fail that match.
>
> Will this force a fallback to direct mod_objcg_mlstate() updates on every
> object allocation and free, causing severe cacheline contention?
This is not that worrying. Yes, the stats update will take slower path. This can
be fixed when we add multiple objcg per-cpu stock.
>
> --
> Sashiko AI review · https://sashiko.dev/#/patchset/20260515171953.2224503-1-shakeel.butt@linux.dev?part=1
>
^ permalink raw reply
* Re: [PATCHSET cgroup/for-7.2] cgroup: Per-css kill_css_finish deferral
From: Tejun Heo @ 2026-05-15 17:28 UTC (permalink / raw)
To: Johannes Weiner, Michal Koutný
Cc: Sebastian Andrzej Siewior, Petr Malat, Bert Karwatzki,
kernel test robot, Martin Pitt, cgroups, linux-kernel
In-Reply-To: <20260505005121.1230198-1-tj@kernel.org>
Hello,
> Tejun Heo (5):
> cgroup: Inline cgroup_has_tasks() in cgroup.h
> cgroup: Annotate unlocked nr_populated_* accesses with READ_ONCE/WRITE_ONCE
> cgroup: Move populated counters to cgroup_subsys_state
> cgroup: Add per-subsys-css kill_css_finish deferral
> cgroup: Defer kill_css_finish() in cgroup_apply_control_disable()
Applied 1-5 to cgroup/for-7.2.
Thanks.
--
tejun
^ permalink raw reply
* [PATCH] memcg: cache obj_stock by memcg, not by objcg pointer
From: Shakeel Butt @ 2026-05-15 17:19 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA
node, but the per-CPU obj_stock_pcp still keys cached_objcg by
pointer. Cross-NUMA workloads now see a drain on every refill and a
miss on every consume that targets a sibling per-node objcg of the
same memcg, producing the 67.7% stress-ng switch-mq regression
reported by LKP.
stock->nr_bytes are fungible across per-node objcgs of one memcg:
drain_obj_stock() and obj_cgroup_uncharge_pages() both account via
obj_cgroup_memcg(). Treat the cache as keyed by memcg in both
__consume_obj_stock() and __refill_obj_stock() so siblings share the
reserve -- eliminating the drain on free and keeping the alloc fast
path in consume.
Though kernel test robot reported the regression but it was not easy to
reproduce locally. Qi implemented [1] a specialized reproducer to show
the corner case which cause the regression and then Qi tested the patch
and reported that the corner case is eliminated after the patch.
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Link: https://lore.kernel.org/19693be6-7132-446e-b3fc-b7e9f56e5949@linux.dev/ [1]
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Debugged-by: Qi Zheng <qi.zheng@linux.dev>
Tested-by: Qi Zheng <qi.zheng@linux.dev>
---
mm/memcontrol.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d978e18b9b2d..66448f428531 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3210,7 +3210,11 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock,
unsigned int nr_bytes)
{
- if (objcg == READ_ONCE(stock->cached_objcg) &&
+ struct obj_cgroup *cached = READ_ONCE(stock->cached_objcg);
+
+ /* Cache is keyed by memcg; sibling per-node objcgs share the reserve. */
+ if ((cached == objcg ||
+ (cached && obj_cgroup_memcg(cached) == obj_cgroup_memcg(objcg))) &&
stock->nr_bytes >= nr_bytes) {
stock->nr_bytes -= nr_bytes;
return true;
@@ -3318,6 +3322,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
unsigned int nr_bytes,
bool allow_uncharge)
{
+ struct obj_cgroup *cached;
unsigned int nr_pages = 0;
if (!stock) {
@@ -3327,7 +3332,10 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
goto out;
}
- if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
+ cached = READ_ONCE(stock->cached_objcg);
+ /* Same memcg: bytes are fungible, no drain needed. */
+ if (cached != objcg &&
+ (!cached || obj_cgroup_memcg(cached) != obj_cgroup_memcg(objcg))) {
drain_obj_stock(stock);
obj_cgroup_get(objcg);
stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
--
2.53.0-Meta
^ permalink raw reply related
* Re: [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: Shakeel Butt @ 2026-05-15 17:09 UTC (permalink / raw)
To: Qi Zheng
Cc: kernel test robot, oe-lkp, lkp, linux-kernel, Andrew Morton,
David Carlier, Allen Pais, Axel Rasmussen, Baoquan He,
Chengming Zhou, Chen Ridong, David Hildenbrand, Hamza Mahfooz,
Harry Yoo, Hugh Dickins, Imran Khan, Johannes Weiner,
Kamalesh Babulal, Lance Yang, Liam Howlett, Lorenzo Stoakes,
Michal Hocko, Michal Koutný, Mike Rapoport, Muchun Song,
Muchun Song, Nhat Pham, Roman Gushchin, Suren Baghdasaryan,
Usama Arif, Vlastimil Babka, Wei Xu, Yosry Ahmed, Yuanchu Xie,
Zi Yan, Usama Arif, cgroups, linux-mm
In-Reply-To: <19693be6-7132-446e-b3fc-b7e9f56e5949@linux.dev>
On Fri, May 15, 2026 at 03:37:22PM +0800, Qi Zheng wrote:
> Hi Shakeel,
>
> On 5/14/26 9:40 PM, Shakeel Butt wrote:
> > May 14, 2026 at 12:46 AM, "Qi Zheng" <qi.zheng@linux.dev mailto:qi.zheng@linux.dev?to=%22Qi%20Zheng%22%20%3Cqi.zheng%40linux.dev%3E > wrote:
> >
> >
> > >
> > > On 5/13/26 10:27 PM, Shakeel Butt wrote:
> > >
> > > >
> > > > On Wed, May 13, 2026 at 06:49:45AM -0700, Shakeel Butt wrote:
> > > >
> > > > >
> > > > > On Wed, May 13, 2026 at 10:10:34AM +0800, Qi Zheng wrote:
> > > > >
> > > > On 5/13/26 12:03 AM, Shakeel Butt wrote:
> > > > On Tue, May 12, 2026 at 08:56:52PM +0800, kernel test robot wrote:
> > > >
> > > > Hello,
> > > >
> > > > kernel test robot noticed a 67.7% regression of stress-ng.switch.ops_per_sec on:
> > > >
> > > > commit: 01b9da291c4969354807b52956f4aae1f41b4924 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > >
> > > > This is most probably due to shuffling of struct mem_cgroup and struct
> > > > mem_cgroup_per_node members.
> > > >
> > > > Another possibility is that after objcg was split into per-node, the
> > > > slab accounting fast path is still designed assuming only one current
> > > > objcg per CPU:
> > > >
> > > > struct obj_stock_pcp {
> > > > struct obj_cgroup *cached_objcg;
> > > > };
> > > >
> > > > So it's may cause the following thrashing:
> > > >
> > > > CPU stock cached = memcg/node0 objcg
> > > > free object tagged = memcg/node1 objcg
> > > > => __refill_obj_stock --> objcg mismatch
> > > > => drain_obj_stock()
> > > > => cache switches to node1 objcg
> > > >
> > > > next local allocation tagged = node0 objcg
> > > > => mismatch again
> > > > => drain_obj_stock()
> > > >
> > > > >
> > > > > Actually I think this is the issue, we have ping pong threads running on
> > > > > different nodes where though theu are in same cgroup but their current->obcg is
> > > > > for local node and thus this ping pong is thrashing the per-cpu objcg stock.
> > > > >
> > > > > The easier fix would be to compare objcg->memcg instead of just objcg during
> > > > > draining and caching. In addition we can add support for multiple objcg per-cpu
> > > > > stock caching.
> > > > >
> > > > Something like the following:
> > > > From d756abe831a905d6fe32bad9a984fc619dafb7e0 Mon Sep 17 00:00:00 2001
> > > > From: Shakeel Butt <shakeel.butt@linux.dev>
> > > > Date: Wed, 13 May 2026 07:24:55 -0700
> > > > Subject: [PATCH] mm/memcontrol: skip obj_stock drain when refilled objcg
> > > > shares memcg
> > > > Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> > > > ---
> > > > mm/memcontrol.c | 14 +++++++++++++-
> > > > 1 file changed, 13 insertions(+), 1 deletion(-)
> > > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > > index d978e18b9b2d..01ed7a8e18ac 100644
> > > > --- a/mm/memcontrol.c
> > > > +++ b/mm/memcontrol.c
> > > > @@ -3318,6 +3318,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > unsigned int nr_bytes,
> > > > bool allow_uncharge)
> > > > {
> > > > + struct obj_cgroup *cached;
> > > > unsigned int nr_pages = 0;
> > > > > if (!stock) {
> > > > @@ -3327,7 +3328,18 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > goto out;
> > > > }
> > > > > - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> > > > + cached = READ_ONCE(stock->cached_objcg);
> > > > + if (cached != objcg &&
> > > > + (!cached || obj_cgroup_memcg(cached) != obj_cgroup_memcg(objcg))) {
> > > > drain_obj_stock(stock);
> > > > obj_cgroup_get(objcg);
> > > > stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> > > >
> > > This change looks like it should be able to fix the ping-pong issue, but
> > > I stiil haven't reproduced the performance regression locally. I'll
> > > continue testing it.
> >
> > Same here, couldn't reproduce locally. It seems like we had to craft a scenario
> > where the pair pingpong threads get their current->objcg from different nodes.
> > I will try that.
>
> I still haven't been able to reproduce the LKP results locally, but I
> used an AI bot to generate a pingpong test case (pasted at the end) and
> automatically ran the test on a physical machine. The results are as
> follows:
>
> parent: 8285917d6f
> bad: 01b9da291c
> fix: 01b9da291c + stock patch
>
> | kernel | mq_ops/sec mean | vs parent | drain_obj_stock / round |
> |--------|-----------------|-----------|-------------------------|
> | parent | 9.743M | baseline | ~0 |
> | bad | 7.821M | -19.73% | ~11.16M |
> | fix | 9.274M | -4.81% | ~0 |
>
> Probing the drain_obj_stock() calls confirms that the fix restores the
> frequency to the parent's baseline.
>
> And it seems that besides __refill_obj_stock(), we should also modify
> __consume_obj_stock()?
>
Thanks a lot Qi. I will send the formal patch and will add your Debugged-by if
you don't mind.
^ permalink raw reply
* Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: T.J. Mercier @ 2026-05-15 17:06 UTC (permalink / raw)
To: Christian Brauner
Cc: Albert Esteve, Tejun Heo, Johannes Weiner, Michal Koutný,
Jonathan Corbet, Shuah Khan, Sumit Semwal, Christian König,
Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, Benjamin Gaignard, Brian Starkey, John Stultz,
Paul Moore, James Morris, Serge E. Hallyn, Stephen Smalley,
Ondrej Mosnacek, Shuah Khan, cgroups, linux-doc, linux-kernel,
linux-media, dri-devel, linaro-mm-sig, linux-mm,
linux-security-module, selinux, linux-kselftest, mripard,
echanude
In-Reply-To: <20260515-hinschauen-effizient-9e3a05a94f2e@brauner>
On Fri, May 15, 2026 at 6:53 AM Christian Brauner <brauner@kernel.org> wrote:
>
> On Tue, May 12, 2026 at 11:10:44AM +0200, Albert Esteve wrote:
> > On embedded platforms a central process often allocates dma-buf
> > memory on behalf of client applications. Without a way to
> > attribute the charge to the requesting client's cgroup, the
> > cost lands on the allocator, making per-cgroup memory limits
> > ineffective for the actual consumers.
> >
> > Add charge_pid_fd to struct dma_heap_allocation_data. When set to
>
> Please be aware that pidfds come in two flavors:
>
> thread-group pidfds and thread-specific pidfds. Make sure that your API
> doesn't implicitly depend on this distinction not existing.
Hi Christian,
Memcg is not a controller that supports "thread mode" so all threads
in a group should belong to the same memcg.
Checking the flags from pidfd_get_pid would be the best way for an
explicit check of the pidfd type?
> > a valid pidfd, DMA_HEAP_IOCTL_ALLOC resolves the target task's
> > memcg and charges the buffer there via mem_cgroup_charge_dmabuf()
> > inside dma_heap_buffer_alloc(). Without charge_pid_fd, and with
> > the mem_accounting module parameter enabled, the buffer is charged
> > to the allocator's own cgroup.
> >
> > Additionally, commit 3c227be90659 ("dma-buf: system_heap: account for
> > system heap allocation in memcg") adds __GFP_ACCOUNT to system-heap
> > page allocations. Keeping __GFP_ACCOUNT would charge the same pages
> > twice (once to kmem, once to MEMCG_DMABUF), thus remove it and route
> > all accounting through a single MEMCG_DMABUF path.
> >
> > Usage examples:
> >
> > 1. Central allocator charging to a client at allocation time.
> > The allocator knows the client's PID (e.g., from binder's
> > sender_pid) and uses pidfd to attribute the charge:
> >
> > pid_t client_pid = txn->sender_pid;
> > int pidfd = pidfd_open(client_pid, 0);
> >
> > struct dma_heap_allocation_data alloc = {
> > .len = buffer_size,
> > .fd_flags = O_RDWR | O_CLOEXEC,
> > .charge_pid_fd = pidfd,
> > };
> > ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> > close(pidfd);
> > /* alloc.fd is now charged to client's cgroup */
> >
> > 2. Default allocation (no pidfd, mem_accounting=1).
> > When charge_pid_fd is not set and the mem_accounting module
> > parameter is enabled, the buffer is charged to the allocator's
> > own cgroup:
> >
> > struct dma_heap_allocation_data alloc = {
> > .len = buffer_size,
> > .fd_flags = O_RDWR | O_CLOEXEC,
> > };
> > ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> > /* charged to current process's cgroup */
> >
> > Current limitations:
> >
> > - Single-owner model: a dma-buf carries one memcg charge regardless of
> > how many processes share it. Means only the first owner (and exporter)
> > of the shared buffer bears the charge.
> > - Only memcg accounting supported. While this makes sense for system
> > heap buffers, other heaps (e.g., CMA heaps) will require selectively
> > charging also for the dmem controller.
> >
> > Signed-off-by: Albert Esteve <aesteve@redhat.com>
> > ---
> > Documentation/admin-guide/cgroup-v2.rst | 5 ++--
> > drivers/dma-buf/dma-buf.c | 16 ++++---------
> > drivers/dma-buf/dma-heap.c | 42 ++++++++++++++++++++++++++++++---
> > drivers/dma-buf/heaps/system_heap.c | 2 --
> > include/uapi/linux/dma-heap.h | 6 +++++
> > 5 files changed, 53 insertions(+), 18 deletions(-)
> >
> > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> > index 8bdbc2e866430..824d269531eb1 100644
> > --- a/Documentation/admin-guide/cgroup-v2.rst
> > +++ b/Documentation/admin-guide/cgroup-v2.rst
> > @@ -1636,8 +1636,9 @@ The following nested keys are defined.
> > structures.
> >
> > dmabuf (npn)
> > - Amount of memory used for exported DMA buffers allocated by the cgroup.
> > - Stays with the allocating cgroup regardless of how the buffer is shared.
> > + Amount of memory used for exported DMA buffers allocated by or on
> > + behalf of the cgroup. Stays with the allocating cgroup regardless
> > + of how the buffer is shared.
> >
> > workingset_refault_anon
> > Number of refaults of previously evicted anonymous pages.
> > diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
> > index ce02377f48908..23fb758b78297 100644
> > --- a/drivers/dma-buf/dma-buf.c
> > +++ b/drivers/dma-buf/dma-buf.c
> > @@ -181,8 +181,11 @@ static void dma_buf_release(struct dentry *dentry)
> > */
> > BUG_ON(dmabuf->cb_in.active || dmabuf->cb_out.active);
> >
> > - mem_cgroup_uncharge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
> > - mem_cgroup_put(dmabuf->memcg);
> > + if (dmabuf->memcg) {
> > + mem_cgroup_uncharge_dmabuf(dmabuf->memcg,
> > + PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
> > + mem_cgroup_put(dmabuf->memcg);
> > + }
> >
> > dmabuf->ops->release(dmabuf);
> >
> > @@ -764,13 +767,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
> > dmabuf->resv = resv;
> > }
> >
> > - dmabuf->memcg = get_mem_cgroup_from_mm(current->mm);
> > - if (!mem_cgroup_charge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE,
> > - GFP_KERNEL)) {
> > - ret = -ENOMEM;
> > - goto err_memcg;
> > - }
> > -
> > file->private_data = dmabuf;
> > file->f_path.dentry->d_fsdata = dmabuf;
> > dmabuf->file = file;
> > @@ -781,8 +777,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
> >
> > return dmabuf;
> >
> > -err_memcg:
> > - mem_cgroup_put(dmabuf->memcg);
> > err_file:
> > fput(file);
> > err_module:
> > diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
> > index ac5f8685a6494..ff6e259afcdc0 100644
> > --- a/drivers/dma-buf/dma-heap.c
> > +++ b/drivers/dma-buf/dma-heap.c
> > @@ -7,13 +7,17 @@
> > */
> >
> > #include <linux/cdev.h>
> > +#include <linux/cgroup.h>
> > #include <linux/device.h>
> > #include <linux/dma-buf.h>
> > #include <linux/dma-heap.h>
> > +#include <linux/memcontrol.h>
> > +#include <linux/sched/mm.h>
> > #include <linux/err.h>
> > #include <linux/export.h>
> > #include <linux/list.h>
> > #include <linux/nospec.h>
> > +#include <linux/pidfd.h>
> > #include <linux/syscalls.h>
> > #include <linux/uaccess.h>
> > #include <linux/xarray.h>
> > @@ -55,10 +59,12 @@ MODULE_PARM_DESC(mem_accounting,
> > "Enable cgroup-based memory accounting for dma-buf heap allocations (default=false).");
> >
> > static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
> > - u32 fd_flags,
> > - u64 heap_flags)
> > + u32 fd_flags, u64 heap_flags,
> > + struct mem_cgroup *charge_to)
> > {
> > struct dma_buf *dmabuf;
> > + unsigned int nr_pages;
> > + struct mem_cgroup *memcg = charge_to;
> > int fd;
> >
> > /*
> > @@ -73,6 +79,22 @@ static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
> > if (IS_ERR(dmabuf))
> > return PTR_ERR(dmabuf);
> >
> > + nr_pages = len / PAGE_SIZE;
> > +
> > + if (memcg)
> > + css_get(&memcg->css);
> > + else if (mem_accounting)
> > + memcg = get_mem_cgroup_from_mm(current->mm);
> > +
> > + if (memcg) {
> > + if (!mem_cgroup_charge_dmabuf(memcg, nr_pages, GFP_KERNEL)) {
> > + mem_cgroup_put(memcg);
> > + dma_buf_put(dmabuf);
> > + return -ENOMEM;
> > + }
> > + dmabuf->memcg = memcg;
> > + }
> > +
> > fd = dma_buf_fd(dmabuf, fd_flags);
> > if (fd < 0) {
> > dma_buf_put(dmabuf);
> > @@ -102,6 +124,9 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
> > {
> > struct dma_heap_allocation_data *heap_allocation = data;
> > struct dma_heap *heap = file->private_data;
> > + struct mem_cgroup *memcg = NULL;
> > + struct task_struct *task;
> > + unsigned int pidfd_flags;
> > int fd;
> >
> > if (heap_allocation->fd)
> > @@ -113,9 +138,20 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
> > if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS)
> > return -EINVAL;
> >
> > + if (heap_allocation->charge_pid_fd) {
> > + task = pidfd_get_task(heap_allocation->charge_pid_fd, &pidfd_flags);
>
> Will always get a thread-group leader pidfd and will fail if this is a
> thread-specific pidfd. pidfd_open(1234, PIDFD_THREAD) can be used to
> open a thread-specific pidfd.
>
> > + if (IS_ERR(task))
> > + return PTR_ERR(task);
> > +
> > + memcg = get_mem_cgroup_from_mm(task->mm);
> > + put_task_struct(task);
> > + }
> > +
> > fd = dma_heap_buffer_alloc(heap, heap_allocation->len,
> > heap_allocation->fd_flags,
> > - heap_allocation->heap_flags);
> > + heap_allocation->heap_flags,
> > + memcg);
> > + mem_cgroup_put(memcg);
> > if (fd < 0)
> > return fd;
> >
> > diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> > index 03c2b87cb1112..95d7688167b93 100644
> > --- a/drivers/dma-buf/heaps/system_heap.c
> > +++ b/drivers/dma-buf/heaps/system_heap.c
> > @@ -385,8 +385,6 @@ static struct page *alloc_largest_available(unsigned long size,
> > if (max_order < orders[i])
> > continue;
> > flags = order_flags[i];
> > - if (mem_accounting)
> > - flags |= __GFP_ACCOUNT;
> > page = alloc_pages(flags, orders[i]);
> > if (!page)
> > continue;
> > diff --git a/include/uapi/linux/dma-heap.h b/include/uapi/linux/dma-heap.h
> > index a4cf716a49fa6..e02b0f8cbc6a1 100644
> > --- a/include/uapi/linux/dma-heap.h
> > +++ b/include/uapi/linux/dma-heap.h
> > @@ -29,6 +29,10 @@
> > * handle to the allocated dma-buf
> > * @fd_flags: file descriptor flags used when allocating
> > * @heap_flags: flags passed to heap
> > + * @charge_pid_fd: optional pidfd of the process whose cgroup should be
> > + * charged for this allocation; 0 means charge the calling
> > + * process's cgroup
> > + * @__padding: reserved, must be zero
> > *
> > * Provided by userspace as an argument to the ioctl
> > */
> > @@ -37,6 +41,8 @@ struct dma_heap_allocation_data {
> > __u32 fd;
> > __u32 fd_flags;
> > __u64 heap_flags;
> > + __u32 charge_pid_fd;
> > + __u32 __padding;
> > };
> >
> > #define DMA_HEAP_IOC_MAGIC 'H'
> >
> > --
> > 2.53.0
> >
^ permalink raw reply
* Re: [PATCH] cgroup/rstat: validate cpu before css_rstat_cpu() access
From: Tejun Heo @ 2026-05-15 16:27 UTC (permalink / raw)
To: Qing Ming; +Cc: Johannes Weiner, Michal Koutný, cgroups, bpf, linux-kernel
In-Reply-To: <20260515122952.59209-1-a0yami@mailbox.org>
On Fri, May 15, 2026 at 08:29:52PM +0800, Qing Ming wrote:
> @@ -90,6 +91,9 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
> !IS_ENABLED(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS)) && in_nmi())
> return;
>
> + if (unlikely(cpu < 0 || cpu >= nr_cpu_ids || !cpu_possible(cpu)))
> + return;
> +
Can you please add this validation to the BPF kfunc that's exposing it?
Thanks.
--
tejun
^ permalink raw reply
* [tj-cgroup:for-next] BUILD SUCCESS 732823fa1fe926424401bd71d63c8cd49738d67e
From: kernel test robot @ 2026-05-15 14:09 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next
branch HEAD: 732823fa1fe926424401bd71d63c8cd49738d67e Merge branch 'for-7.2' into for-next
elapsed time: 983m
configs tested: 216
configs skipped: 4
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allmodconfig gcc-15.2.0
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc defconfig gcc-15.2.0
arc hsdk_defconfig gcc-15.2.0
arc randconfig-001 gcc-8.5.0
arc randconfig-001-20260515 gcc-8.5.0
arc randconfig-002 gcc-8.5.0
arc randconfig-002-20260515 gcc-8.5.0
arm allnoconfig clang-23
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm allyesconfig gcc-15.2.0
arm defconfig gcc-15.2.0
arm randconfig-001 gcc-8.5.0
arm randconfig-001-20260515 gcc-8.5.0
arm randconfig-002 gcc-8.5.0
arm randconfig-002-20260515 gcc-8.5.0
arm randconfig-003 gcc-8.5.0
arm randconfig-003-20260515 gcc-8.5.0
arm randconfig-004 gcc-8.5.0
arm randconfig-004-20260515 gcc-8.5.0
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001-20260515 clang-16
arm64 randconfig-001-20260515 gcc-11.5.0
arm64 randconfig-002-20260515 gcc-10.5.0
arm64 randconfig-002-20260515 gcc-11.5.0
arm64 randconfig-003-20260515 gcc-11.5.0
arm64 randconfig-004-20260515 gcc-11.5.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001-20260515 gcc-10.5.0
csky randconfig-001-20260515 gcc-11.5.0
csky randconfig-002-20260515 gcc-11.5.0
csky randconfig-002-20260515 gcc-15.2.0
hexagon allmodconfig clang-17
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig clang-23
hexagon allnoconfig gcc-15.2.0
hexagon defconfig gcc-15.2.0
i386 allmodconfig clang-20
i386 allmodconfig gcc-14
i386 allnoconfig gcc-14
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 allyesconfig gcc-14
i386 buildonly-randconfig-001 gcc-14
i386 buildonly-randconfig-001-20260515 gcc-14
i386 buildonly-randconfig-002 gcc-14
i386 buildonly-randconfig-002-20260515 gcc-14
i386 buildonly-randconfig-003 gcc-14
i386 buildonly-randconfig-003-20260515 gcc-14
i386 buildonly-randconfig-004 gcc-14
i386 buildonly-randconfig-004-20260515 gcc-14
i386 buildonly-randconfig-005 gcc-14
i386 buildonly-randconfig-005-20260515 gcc-14
i386 buildonly-randconfig-006 gcc-14
i386 buildonly-randconfig-006-20260515 gcc-14
i386 defconfig gcc-15.2.0
i386 randconfig-001 clang-20
i386 randconfig-001-20260515 clang-20
i386 randconfig-002 clang-20
i386 randconfig-002-20260515 clang-20
i386 randconfig-003 clang-20
i386 randconfig-003-20260515 clang-20
i386 randconfig-004 clang-20
i386 randconfig-004-20260515 clang-20
i386 randconfig-005 clang-20
i386 randconfig-005-20260515 clang-20
i386 randconfig-006 clang-20
i386 randconfig-006-20260515 clang-20
i386 randconfig-007 clang-20
i386 randconfig-007-20260515 clang-20
i386 randconfig-011 gcc-14
i386 randconfig-011-20260515 gcc-14
i386 randconfig-012 gcc-14
i386 randconfig-012-20260515 gcc-14
i386 randconfig-013 gcc-14
i386 randconfig-013-20260515 gcc-14
i386 randconfig-014 gcc-14
i386 randconfig-014-20260515 gcc-14
i386 randconfig-015 gcc-14
i386 randconfig-015-20260515 gcc-14
i386 randconfig-016 gcc-14
i386 randconfig-016-20260515 gcc-14
i386 randconfig-017 gcc-14
i386 randconfig-017-20260515 gcc-14
loongarch allmodconfig clang-23
loongarch allnoconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k allyesconfig gcc-15.2.0
m68k defconfig clang-19
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
nios2 allmodconfig clang-23
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig clang-23
nios2 allnoconfig gcc-11.5.0
nios2 defconfig clang-19
openrisc allmodconfig clang-23
openrisc allnoconfig clang-23
openrisc allnoconfig gcc-15.2.0
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allnoconfig gcc-15.2.0
parisc allyesconfig clang-19
parisc allyesconfig gcc-15.2.0
parisc defconfig gcc-15.2.0
parisc randconfig-001-20260515 gcc-8.5.0
parisc randconfig-002-20260515 gcc-8.5.0
parisc64 defconfig clang-19
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc allnoconfig gcc-15.2.0
powerpc randconfig-001-20260515 gcc-8.5.0
powerpc randconfig-002-20260515 gcc-8.5.0
powerpc64 randconfig-001-20260515 gcc-8.5.0
powerpc64 randconfig-002-20260515 gcc-8.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allnoconfig gcc-15.2.0
riscv allyesconfig clang-16
riscv defconfig gcc-15.2.0
riscv randconfig-001-20260515 gcc-15.2.0
riscv randconfig-002-20260515 gcc-15.2.0
s390 allmodconfig clang-18
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig gcc-15.2.0
s390 randconfig-001-20260515 gcc-15.2.0
s390 randconfig-002-20260515 gcc-15.2.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allnoconfig gcc-15.2.0
sh allyesconfig clang-19
sh allyesconfig gcc-15.2.0
sh defconfig gcc-14
sh randconfig-001-20260515 gcc-15.2.0
sh randconfig-002-20260515 gcc-15.2.0
sparc allnoconfig clang-23
sparc allnoconfig gcc-15.2.0
sparc defconfig gcc-15.2.0
sparc randconfig-001-20260515 gcc-8.5.0
sparc randconfig-002-20260515 gcc-8.5.0
sparc64 allmodconfig clang-23
sparc64 defconfig gcc-14
sparc64 randconfig-001-20260515 gcc-8.5.0
sparc64 randconfig-002-20260515 gcc-8.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-14
um allyesconfig gcc-15.2.0
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001-20260515 gcc-8.5.0
um randconfig-002-20260515 gcc-8.5.0
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20260515 gcc-14
x86_64 buildonly-randconfig-002-20260515 gcc-14
x86_64 buildonly-randconfig-003-20260515 gcc-14
x86_64 buildonly-randconfig-004-20260515 gcc-14
x86_64 buildonly-randconfig-005-20260515 gcc-14
x86_64 buildonly-randconfig-006-20260515 gcc-14
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-011-20260515 clang-20
x86_64 randconfig-012-20260515 clang-20
x86_64 randconfig-013-20260515 clang-20
x86_64 randconfig-014-20260515 clang-20
x86_64 randconfig-015-20260515 clang-20
x86_64 randconfig-016-20260515 clang-20
x86_64 randconfig-071 gcc-12
x86_64 randconfig-071-20260515 gcc-12
x86_64 randconfig-072 gcc-12
x86_64 randconfig-072-20260515 gcc-12
x86_64 randconfig-073 gcc-12
x86_64 randconfig-073-20260515 gcc-12
x86_64 randconfig-074 gcc-12
x86_64 randconfig-074-20260515 gcc-12
x86_64 randconfig-075 gcc-12
x86_64 randconfig-075-20260515 gcc-12
x86_64 randconfig-076 gcc-12
x86_64 randconfig-076-20260515 gcc-12
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allnoconfig gcc-15.2.0
xtensa allyesconfig clang-23
xtensa randconfig-001-20260515 gcc-8.5.0
xtensa randconfig-002-20260515 gcc-8.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Christian Brauner @ 2026-05-15 13:53 UTC (permalink / raw)
To: Albert Esteve
Cc: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Paul Moore, James Morris, Serge E. Hallyn, Stephen Smalley,
Ondrej Mosnacek, Shuah Khan, cgroups, linux-doc, linux-kernel,
linux-media, dri-devel, linaro-mm-sig, linux-mm,
linux-security-module, selinux, linux-kselftest, mripard,
echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-2-6326701c3691@redhat.com>
On Tue, May 12, 2026 at 11:10:44AM +0200, Albert Esteve wrote:
> On embedded platforms a central process often allocates dma-buf
> memory on behalf of client applications. Without a way to
> attribute the charge to the requesting client's cgroup, the
> cost lands on the allocator, making per-cgroup memory limits
> ineffective for the actual consumers.
>
> Add charge_pid_fd to struct dma_heap_allocation_data. When set to
Please be aware that pidfds come in two flavors:
thread-group pidfds and thread-specific pidfds. Make sure that your API
doesn't implicitly depend on this distinction not existing.
> a valid pidfd, DMA_HEAP_IOCTL_ALLOC resolves the target task's
> memcg and charges the buffer there via mem_cgroup_charge_dmabuf()
> inside dma_heap_buffer_alloc(). Without charge_pid_fd, and with
> the mem_accounting module parameter enabled, the buffer is charged
> to the allocator's own cgroup.
>
> Additionally, commit 3c227be90659 ("dma-buf: system_heap: account for
> system heap allocation in memcg") adds __GFP_ACCOUNT to system-heap
> page allocations. Keeping __GFP_ACCOUNT would charge the same pages
> twice (once to kmem, once to MEMCG_DMABUF), thus remove it and route
> all accounting through a single MEMCG_DMABUF path.
>
> Usage examples:
>
> 1. Central allocator charging to a client at allocation time.
> The allocator knows the client's PID (e.g., from binder's
> sender_pid) and uses pidfd to attribute the charge:
>
> pid_t client_pid = txn->sender_pid;
> int pidfd = pidfd_open(client_pid, 0);
>
> struct dma_heap_allocation_data alloc = {
> .len = buffer_size,
> .fd_flags = O_RDWR | O_CLOEXEC,
> .charge_pid_fd = pidfd,
> };
> ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> close(pidfd);
> /* alloc.fd is now charged to client's cgroup */
>
> 2. Default allocation (no pidfd, mem_accounting=1).
> When charge_pid_fd is not set and the mem_accounting module
> parameter is enabled, the buffer is charged to the allocator's
> own cgroup:
>
> struct dma_heap_allocation_data alloc = {
> .len = buffer_size,
> .fd_flags = O_RDWR | O_CLOEXEC,
> };
> ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> /* charged to current process's cgroup */
>
> Current limitations:
>
> - Single-owner model: a dma-buf carries one memcg charge regardless of
> how many processes share it. Means only the first owner (and exporter)
> of the shared buffer bears the charge.
> - Only memcg accounting supported. While this makes sense for system
> heap buffers, other heaps (e.g., CMA heaps) will require selectively
> charging also for the dmem controller.
>
> Signed-off-by: Albert Esteve <aesteve@redhat.com>
> ---
> Documentation/admin-guide/cgroup-v2.rst | 5 ++--
> drivers/dma-buf/dma-buf.c | 16 ++++---------
> drivers/dma-buf/dma-heap.c | 42 ++++++++++++++++++++++++++++++---
> drivers/dma-buf/heaps/system_heap.c | 2 --
> include/uapi/linux/dma-heap.h | 6 +++++
> 5 files changed, 53 insertions(+), 18 deletions(-)
>
> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> index 8bdbc2e866430..824d269531eb1 100644
> --- a/Documentation/admin-guide/cgroup-v2.rst
> +++ b/Documentation/admin-guide/cgroup-v2.rst
> @@ -1636,8 +1636,9 @@ The following nested keys are defined.
> structures.
>
> dmabuf (npn)
> - Amount of memory used for exported DMA buffers allocated by the cgroup.
> - Stays with the allocating cgroup regardless of how the buffer is shared.
> + Amount of memory used for exported DMA buffers allocated by or on
> + behalf of the cgroup. Stays with the allocating cgroup regardless
> + of how the buffer is shared.
>
> workingset_refault_anon
> Number of refaults of previously evicted anonymous pages.
> diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
> index ce02377f48908..23fb758b78297 100644
> --- a/drivers/dma-buf/dma-buf.c
> +++ b/drivers/dma-buf/dma-buf.c
> @@ -181,8 +181,11 @@ static void dma_buf_release(struct dentry *dentry)
> */
> BUG_ON(dmabuf->cb_in.active || dmabuf->cb_out.active);
>
> - mem_cgroup_uncharge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
> - mem_cgroup_put(dmabuf->memcg);
> + if (dmabuf->memcg) {
> + mem_cgroup_uncharge_dmabuf(dmabuf->memcg,
> + PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
> + mem_cgroup_put(dmabuf->memcg);
> + }
>
> dmabuf->ops->release(dmabuf);
>
> @@ -764,13 +767,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
> dmabuf->resv = resv;
> }
>
> - dmabuf->memcg = get_mem_cgroup_from_mm(current->mm);
> - if (!mem_cgroup_charge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE,
> - GFP_KERNEL)) {
> - ret = -ENOMEM;
> - goto err_memcg;
> - }
> -
> file->private_data = dmabuf;
> file->f_path.dentry->d_fsdata = dmabuf;
> dmabuf->file = file;
> @@ -781,8 +777,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
>
> return dmabuf;
>
> -err_memcg:
> - mem_cgroup_put(dmabuf->memcg);
> err_file:
> fput(file);
> err_module:
> diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
> index ac5f8685a6494..ff6e259afcdc0 100644
> --- a/drivers/dma-buf/dma-heap.c
> +++ b/drivers/dma-buf/dma-heap.c
> @@ -7,13 +7,17 @@
> */
>
> #include <linux/cdev.h>
> +#include <linux/cgroup.h>
> #include <linux/device.h>
> #include <linux/dma-buf.h>
> #include <linux/dma-heap.h>
> +#include <linux/memcontrol.h>
> +#include <linux/sched/mm.h>
> #include <linux/err.h>
> #include <linux/export.h>
> #include <linux/list.h>
> #include <linux/nospec.h>
> +#include <linux/pidfd.h>
> #include <linux/syscalls.h>
> #include <linux/uaccess.h>
> #include <linux/xarray.h>
> @@ -55,10 +59,12 @@ MODULE_PARM_DESC(mem_accounting,
> "Enable cgroup-based memory accounting for dma-buf heap allocations (default=false).");
>
> static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
> - u32 fd_flags,
> - u64 heap_flags)
> + u32 fd_flags, u64 heap_flags,
> + struct mem_cgroup *charge_to)
> {
> struct dma_buf *dmabuf;
> + unsigned int nr_pages;
> + struct mem_cgroup *memcg = charge_to;
> int fd;
>
> /*
> @@ -73,6 +79,22 @@ static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
> if (IS_ERR(dmabuf))
> return PTR_ERR(dmabuf);
>
> + nr_pages = len / PAGE_SIZE;
> +
> + if (memcg)
> + css_get(&memcg->css);
> + else if (mem_accounting)
> + memcg = get_mem_cgroup_from_mm(current->mm);
> +
> + if (memcg) {
> + if (!mem_cgroup_charge_dmabuf(memcg, nr_pages, GFP_KERNEL)) {
> + mem_cgroup_put(memcg);
> + dma_buf_put(dmabuf);
> + return -ENOMEM;
> + }
> + dmabuf->memcg = memcg;
> + }
> +
> fd = dma_buf_fd(dmabuf, fd_flags);
> if (fd < 0) {
> dma_buf_put(dmabuf);
> @@ -102,6 +124,9 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
> {
> struct dma_heap_allocation_data *heap_allocation = data;
> struct dma_heap *heap = file->private_data;
> + struct mem_cgroup *memcg = NULL;
> + struct task_struct *task;
> + unsigned int pidfd_flags;
> int fd;
>
> if (heap_allocation->fd)
> @@ -113,9 +138,20 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
> if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS)
> return -EINVAL;
>
> + if (heap_allocation->charge_pid_fd) {
> + task = pidfd_get_task(heap_allocation->charge_pid_fd, &pidfd_flags);
Will always get a thread-group leader pidfd and will fail if this is a
thread-specific pidfd. pidfd_open(1234, PIDFD_THREAD) can be used to
open a thread-specific pidfd.
> + if (IS_ERR(task))
> + return PTR_ERR(task);
> +
> + memcg = get_mem_cgroup_from_mm(task->mm);
> + put_task_struct(task);
> + }
> +
> fd = dma_heap_buffer_alloc(heap, heap_allocation->len,
> heap_allocation->fd_flags,
> - heap_allocation->heap_flags);
> + heap_allocation->heap_flags,
> + memcg);
> + mem_cgroup_put(memcg);
> if (fd < 0)
> return fd;
>
> diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> index 03c2b87cb1112..95d7688167b93 100644
> --- a/drivers/dma-buf/heaps/system_heap.c
> +++ b/drivers/dma-buf/heaps/system_heap.c
> @@ -385,8 +385,6 @@ static struct page *alloc_largest_available(unsigned long size,
> if (max_order < orders[i])
> continue;
> flags = order_flags[i];
> - if (mem_accounting)
> - flags |= __GFP_ACCOUNT;
> page = alloc_pages(flags, orders[i]);
> if (!page)
> continue;
> diff --git a/include/uapi/linux/dma-heap.h b/include/uapi/linux/dma-heap.h
> index a4cf716a49fa6..e02b0f8cbc6a1 100644
> --- a/include/uapi/linux/dma-heap.h
> +++ b/include/uapi/linux/dma-heap.h
> @@ -29,6 +29,10 @@
> * handle to the allocated dma-buf
> * @fd_flags: file descriptor flags used when allocating
> * @heap_flags: flags passed to heap
> + * @charge_pid_fd: optional pidfd of the process whose cgroup should be
> + * charged for this allocation; 0 means charge the calling
> + * process's cgroup
> + * @__padding: reserved, must be zero
> *
> * Provided by userspace as an argument to the ioctl
> */
> @@ -37,6 +41,8 @@ struct dma_heap_allocation_data {
> __u32 fd;
> __u32 fd_flags;
> __u64 heap_flags;
> + __u32 charge_pid_fd;
> + __u32 __padding;
> };
>
> #define DMA_HEAP_IOC_MAGIC 'H'
>
> --
> 2.53.0
>
^ permalink raw reply
* Re: [PATCH v4 04/12] mm, swap: add support for stable large allocation in swap cache directly
From: Kairui Song @ 2026-05-15 13:46 UTC (permalink / raw)
To: kasong
Cc: linux-mm, Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang,
Barry Song, Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Johannes Weiner, Youngjun Park, Chengming Zhou,
Roman Gushchin, Shakeel Butt, Muchun Song, linux-kernel, cgroups,
Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260515-swap-table-p4-v4-4-f1b49e845a8d@tencent.com>
On Fri, May 15, 2026 at 05:54:17PM +0800, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
>
> To make it possible to allocate large folios directly in swap cache,
> provide a new infrastructure helper to handle the swap cache status
> check, allocation, and order fallback in the swap cache layer
>
> The new helper replaces the existing swap_cache_alloc_folio. Based on
> this, all the separate swap folio allocation that is being done by anon
> / shmem before is converted to use this helper directly, unifying folio
> allocation for anon, shmem, and readahead.
>
> This slightly consolidates how allocation is synchronized, making it
> more stable and less prone to errors. The slot-count and cache-conflict
> check is now always performed with the cluster lock held before
> allocation, and repeated under the same lock right before cache
> insertion. This double check produces a stable result compared to the
> previous anon and shmem mTHP allocation implementation, avoids the
> false-negative conflict checks that the lockless path can return — large
> allocations no longer have to be unwound because the range turned out to
> be occupied — and aborts early for already-freed slots, which helps
> ordinary swapin and especially readahead, with only a marginal increase
> in cluster-lock contention (the lock is very lightly contended and stays
> local in the first place). Hence, callers of swap_cache_alloc_folio() no
> longer need to check the swap slot count or swap cache status
> themselves.
>
> And now whoever first successfully allocates a folio in the swap cache
> will be the one who charges it and performs the swap-in. The race window
> of swapping is also reduced since the loop is much more compact.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> mm/swap.h | 3 +-
> mm/swap_state.c | 234 +++++++++++++++++++++++++++++++++++++++-----------------
> mm/zswap.c | 2 +-
> 3 files changed, 168 insertions(+), 71 deletions(-)
>
> diff --git a/mm/swap.h b/mm/swap.h
> index ad8b17a93758..6774af10a943 100644
> --- a/mm/swap.h
> +++ b/mm/swap.h
> @@ -280,7 +280,8 @@ bool swap_cache_has_folio(swp_entry_t entry);
> struct folio *swap_cache_get_folio(swp_entry_t entry);
> void *swap_cache_get_shadow(swp_entry_t entry);
> void swap_cache_del_folio(struct folio *folio);
> -struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags,
> +struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask,
> + unsigned long orders, struct vm_fault *vmf,
> struct mempolicy *mpol, pgoff_t ilx);
> /* Below helpers require the caller to lock and pass in the swap cluster. */
> void __swap_cache_add_folio(struct swap_cluster_info *ci,
> diff --git a/mm/swap_state.c b/mm/swap_state.c
> index 89fa19ec13f6..cd4543ff5e47 100644
> --- a/mm/swap_state.c
> +++ b/mm/swap_state.c
> @@ -139,10 +139,10 @@ void *swap_cache_get_shadow(swp_entry_t entry)
>
> /**
> * __swap_cache_add_check - Check if a range is suitable for adding a folio.
> - * @ci: The locked swap cluster.
> - * @ci_off: Range start offset.
> - * @nr: Number of slots to check.
> - * @shadow: Returns the shadow value if one exists in the range.
> + * @ci: The locked swap cluster
> + * @targ_entry: The target swap entry to check, will be rounded down by @nr
> + * @nr: Number of slots to check, must be a power of 2
> + * @shadowp: Returns the shadow value if one exists in the range.
> *
> * Check if all slots covered by given range have a swap count >= 1.
> * Retrieves the shadow if there is one.
> @@ -151,26 +151,40 @@ void *swap_cache_get_shadow(swp_entry_t entry)
> * Return: 0 if success, error code if failed.
> */
> static int __swap_cache_add_check(struct swap_cluster_info *ci,
> - unsigned int ci_off, unsigned int nr,
> - void **shadow)
> + swp_entry_t targ_entry,
> + unsigned long nr, void **shadowp)
> {
> - unsigned int ci_end = ci_off + nr;
> + unsigned int ci_off, ci_end;
> unsigned long old_tb;
>
> lockdep_assert_held(&ci->lock);
> - if (WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER))
> - return -EINVAL;
>
> + /*
> + * If the target slot is not swapped out or already cached, return
> + * -ENOENT or -EEXIST. If the batch is not suitable, could be a
> + * race with concurrent free or cache add, return -EBUSY.
> + */
> if (unlikely(!ci->table))
> return -ENOENT;
> + ci_off = swp_cluster_offset(targ_entry);
> + old_tb = __swap_table_get(ci, ci_off);
> + if (swp_tb_is_folio(old_tb))
> + return -EEXIST;
> + if (!__swp_tb_get_count(old_tb))
> + return -ENOENT;
> + if (swp_tb_is_shadow(old_tb) && shadowp)
> + *shadowp = swp_tb_to_shadow(old_tb);
> +
> + if (nr == 1)
> + return 0;
> +
> + ci_off = round_down(ci_off, nr);
> + ci_end = ci_off + nr;
> do {
> old_tb = __swap_table_get(ci, ci_off);
> - if (unlikely(swp_tb_is_folio(old_tb)))
> - return -EEXIST;
> - if (unlikely(!__swp_tb_get_count(old_tb)))
> - return -ENOENT;
> - if (swp_tb_is_shadow(old_tb))
> - *shadow = swp_tb_to_shadow(old_tb);
> + if (unlikely(swp_tb_is_folio(old_tb) ||
> + !__swp_tb_get_count(old_tb)))
> + return -EBUSY;
> } while (++ci_off < ci_end);
>
> return 0;
> @@ -241,15 +255,13 @@ static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,
> {
> int err;
> void *shadow = NULL;
> - unsigned int ci_off;
> struct swap_info_struct *si;
> struct swap_cluster_info *ci;
> unsigned long nr_pages = folio_nr_pages(folio);
>
> si = __swap_entry_to_info(entry);
> ci = swap_cluster_lock(si, swp_offset(entry));
> - ci_off = swp_cluster_offset(entry);
> - err = __swap_cache_add_check(ci, ci_off, nr_pages, &shadow);
> + err = __swap_cache_add_check(ci, entry, nr_pages, &shadow);
> if (err) {
> swap_cluster_unlock(ci);
> return err;
> @@ -404,6 +416,140 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
> }
> }
>
> +/*
> + * Try to allocate a folio of given order in the swap cache.
> + *
> + * This helper resolves the potential races of swap allocation
> + * and prepares a folio to be used for swap IO. May return following
> + * value:
> + *
> + * -ENOMEM / -EBUSY: Order is too large or in conflict with sub slot,
> + * caller should shrink the order and retry
> + * -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
> + * should abort or try to use the cached folio instead
> + */
> +static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
> + swp_entry_t targ_entry, gfp_t gfp,
> + unsigned int order, struct vm_fault *vmf,
> + struct mempolicy *mpol, pgoff_t ilx)
> +{
> + int err;
> + swp_entry_t entry;
> + struct folio *folio;
> + void *shadow = NULL;
> + unsigned long address, nr_pages = 1UL << order;
> + struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
> +
> + VM_WARN_ON_ONCE(nr_pages > SWAPFILE_CLUSTER);
> + entry.val = round_down(targ_entry.val, nr_pages);
> +
> + /* Check if the slot and range are available, skip allocation if not */
> + spin_lock(&ci->lock);
> + err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL);
> + spin_unlock(&ci->lock);
> + if (unlikely(err))
> + return ERR_PTR(err);
> +
> + /*
> + * Limit THP gfp. The limitation is a no-op for typical
> + * GFP_HIGHUSER_MOVABLE but matters for shmem.
> + */
> + if (order)
> + gfp = thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
> +
> + if (mpol || !vmf) {
> + folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
> + } else {
> + address = round_down(vmf->address, PAGE_SIZE << order);
> + folio = vma_alloc_folio(gfp, order, vmf->vma, address);
> + }
> + if (unlikely(!folio))
> + return ERR_PTR(-ENOMEM);
> +
> + /* Double check the range is still not in conflict */
> + spin_lock(&ci->lock);
> + err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow);
> + if (unlikely(err)) {
> + spin_unlock(&ci->lock);
> + folio_put(folio);
> + return ERR_PTR(err);
> + }
> +
> + __folio_set_locked(folio);
> + __folio_set_swapbacked(folio);
> + __swap_cache_do_add_folio(ci, folio, entry);
> + spin_unlock(&ci->lock);
> +
> + if (mem_cgroup_swapin_charge_folio(folio, vmf ? vmf->vma->vm_mm : NULL,
> + gfp, entry)) {
> + spin_lock(&ci->lock);
> + __swap_cache_do_del_folio(ci, folio, entry, shadow);
> + spin_unlock(&ci->lock);
> + folio_unlock(folio);
> + /* nr_pages refs from swap cache, 1 from allocation */
> + folio_put_refs(folio, nr_pages + 1);
> + count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
> + return ERR_PTR(-ENOMEM);
> + }
> +
> + /* For memsw accounting, swap is uncharged when folio is added to swap cache */
> + memcg1_swapin(entry, 1 << order);
> + if (shadow)
> + workingset_refault(folio, shadow);
> +
> + node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
> + lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
> +
> + /* Caller will initiate read into locked new_folio */
> + folio_add_lru(folio);
> + return folio;
> +}
> +
> +/**
> + * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
> + * @targ_entry: swap entry indicating the target slot
> + * @gfp: memory allocation flags
> + * @orders: allocation orders, must be non zero
> + * @vmf: fault information
> + * @mpol: NUMA memory allocation policy to be applied
> + * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
> + *
> + * Allocate a folio in the swap cache for one swap slot, typically before
> + * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
> + * @targ_entry must have a non-zero swap count (swapped out).
> + *
> + * Context: Caller must protect the swap device with reference count or locks.
> + * Return: Returns the folio if allocation succeeded and folio is in the swap
> + * cache. Returns error code if failed due to race, OOM or invalid arguments.
> + */
> +struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
> + unsigned long orders, struct vm_fault *vmf,
> + struct mempolicy *mpol, pgoff_t ilx)
> +{
> + int order, err;
> + struct folio *ret;
> + struct swap_cluster_info *ci;
> +
> + if (WARN_ON_ONCE(!orders))
> + return ERR_PTR(-EINVAL);
> +
> + ci = __swap_entry_to_cluster(targ_entry);
> + order = highest_order(orders);
> + while (orders) {
> + ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
> + vmf, mpol, ilx);
> + if (!IS_ERR(ret))
> + break;
> + err = PTR_ERR(ret);
> + if (err && err != -EBUSY && err != -ENOMEM)
> + break;
> + count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
> + order = next_order(&orders, order);
> + }
I just realized that with !CONFIG_TRANSPARENT_HUGEPAGE,
next_order(&orders, order) won't modify orders so this loop won't
break properly for !CONFIG_TRANSPARENT_HUGEPAGE build.
So V4 is not correct here. I did a "cleanup" since V4 removed
the forced order 0 fallback. The cleanup is wrong. We need to revert
this loop part back to V3 by squashing this:
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 7701fa4b981c..60f93995e492 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -508,13 +508,13 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
ci = __swap_entry_to_cluster(targ_entry);
order = highest_order(orders);
- while (orders) {
+ for (;;) {
ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
vmf, mpol, ilx);
if (!IS_ERR(ret))
break;
err = PTR_ERR(ret);
- if (err && err != -EBUSY && err != -ENOMEM)
+ if (!order || (err && err != -EBUSY && err != -ENOMEM))
break;
count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
order = next_order(&orders, order);
---
Other than that this should be good.
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox