* Re: [PATCH v3 00/12] mm, swap: swap table phase IV: unify allocation and reduce static metadata
From: Kairui Song @ 2026-05-12 5:10 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-mm, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <20260511141249.eac1426fee41c9fe463e7e23@linux-foundation.org>
On Tue, May 12, 2026 at 5:12 AM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Sat, 25 Apr 2026 02:11:47 +0800 Kairui Song <ryncsn@gmail.com> wrote:
> > I checked sashiko's review, it seems sashiko itself is bugged or
> > something wrong, Most patched end up with:
> > Tool error: Review tool timed out (active time exceeded)
> >
> > The rest of the results are all false positives, maybe I can add a few
> > more comments in the code or commit so it can understand the code
> > better.
> >
> > And checking V2's review:
> > https://sashiko.dev/#/patchset/20260417-swap-table-p4-v2-0-17f5d1015428%40tencent.com
> >
> > Which are mostly false positives and I've fixed the two real but
> > trivial issues already. Things should be fine.
>
> Sashiko review of v3:
>
> https://sashiko.dev/#/patchset/20260421-swap-table-p4-v3-0-2f23759a76bc@tencent.com
>
> appears to be complete, so perhaps it went back and figured it out.
>
> It claims to have several "critical" and "high" things, so please
> recheck?
Right, thanks for the head up! Just checked again, still, all reports
are false positives.
Some part may worth adding a WARN_ON or some comment (one was also
suggested by Chris), so both humans and AI will be less confused.
For example sashiko is very concerned about round_down of swp_entry_t,
or alignment of folio's swap entry, which is already a common pattern
now and completely fine. We did plan to use a wrapper for that later
to make it less confusing, not really a problem.
Maybe better also add a bit more info to the commit message.
> From your replies in this thread, I believe that we'll be seeing a v4
> series?
Sure, I'll send a v4, most changes are for cleanup and minor improvements.
^ permalink raw reply
* Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
From: K Prateek Nayak @ 2026-05-12 5:37 UTC (permalink / raw)
To: Peter Zijlstra, mingo
Cc: longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, qyousef
In-Reply-To: <20260511120627.944705718@infradead.org>
Hello Peter,
On 5/11/2026 5:01 PM, Peter Zijlstra wrote:
> @@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_fai
> if (unlikely(throttled))
> task_throttle_setup_work(p);
> return p;
> +
> +idle:
> + new_tasks = sched_balance_newidle(rq, rf);
> + if (new_tasks < 0)
> + return RETRY_TASK;
> + if (new_tasks > 0)
> + goto again;
> + return NULL;
> }
For core scheduling will now trigger a newidle balance during the pick
when core_cookie is reset to 0 which can cause tasks to migrate only
for them to find they cannot run on the CPU since core-wide selection
leads to a cookie mismatch and it is kept hanging there.
Can we return early if sched_core_enabled() here or are the additional
newidle balance okay?
--
Thanks and Regards,
Prateek
^ permalink raw reply
* Re: [PATCH v3 05/12] mm, swap: unify large folio allocation
From: David Hildenbrand (Arm) @ 2026-05-12 6:07 UTC (permalink / raw)
To: Kairui Song
Cc: linux-mm, Andrew Morton, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <CAMgjq7CK7DPfHEXUY+mDeKu33n=CBrh1RKmpem7Arjsas9rxYw@mail.gmail.com>
On 5/11/26 18:44, Kairui Song wrote:
> On Mon, May 11, 2026 at 11:15 PM David Hildenbrand (Arm)
> <david@kernel.org> wrote:
>>
>> On 5/11/26 16:37, Kairui Song wrote:
>>>
>>> Yes, the current status is a bit odd, about two years ago I also
>>> wanted to name it `swapin_direct()`.
>>> https://lore.kernel.org/linux-mm/20240326185032.72159-3-ryncsn@gmail.com/
>>>
>>> But actually ZRAM or shmem would also benefit from supporting unified
>>> readahead like this:
>>> https://lore.kernel.org/linux-mm/20240102175338.62012-6-ryncsn@gmail.com/
>>>
>>> So calling it `swapin_entry` seems more future-proof. At some point in
>>> the future we might remove `swapin_readahead`. All swapin operations
>>> could have a unified or at least a per-device readahead policy like
>>> the one in the link above, instead of the current policy where the
>>> caller must decide whether to perform readahead.
>>>
>>> But any suggestion on naming is welcome :)
>>
>> The other proposal
>>
>> https://lore.kernel.org/all/tencent_CD11FE9B4A0B362E95E776C5F679598FAA07@qq.com/
>>
>> calls it
>>
>> swapin_synchronous_folio
>>
>> Maybe just swapin_sync_io()/swapin_sync() or sth like that?
>
> Good idea, I can keep the swapin_sync name at this point. Sync io flag
> still may remain for a longer time.
BTW, I was also wondering whether the whole sync vs. readahead part could simply
be handled in a function called "swapin". moving that completely out of memory.c :)
Probably something for another cleanup.
--
Cheers,
David
^ permalink raw reply
* [PATCH 0/6] psi: slightly improve performance of psi
From: Luka Bai @ 2026-05-12 6:19 UTC (permalink / raw)
To: linux-mm
Cc: Johannes Weiner, Suren Baghdasaryan, Peter Ziljstra, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak,
Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport, Michal Hocko,
Kees Cook, Tejun Heo, Michal Koutný, linux-kernel, cgroups,
Luka Bai
PSI is useful for resource pressure monitoring. But the callbacks are
distributed among all the common calling paths, some of which are quite
performance critical. The hottest callback like psi_group_change is
called by both psi_task_switch and psi_task_change, which are parts of
task_switch, enqueue, dequeue. So the cpu usage of psi is quite
important.
We initialized a common hackbench test using the following command:
perf record --kernel-callchains -a -g hackbench -s 512 -P -g 10 -f 30 \
-l 1000 --pipe
In a machine setup with 8 cores, 16GB with two numa node(each node 8GB),
we saw a cpu usage of 4.3% for psi using the flame graph of the perf
data, which can make some observable influence to the actual workloads.
In this patchset, we did some improvement for the performance of hot
path, which slightly improves the performance for the psi. With a same
setup of 8 cores + 16GB, the cpu usage of psi becomes 3.4%, which has
a 20% improvement. In the future patches we may try to do more
adjustment to go further (Like add switches for different types of PSI
resources maybe).
Patch Details:
========
* Patch 1 moves the judgement of cpu_curr(cpu)->in_memstall from
psi_group_change outside to eliminate some repeated memory access.
* Patch 2 adds a bit variable need_psi to help judge whether we need
to do psi accouting for the cgroup. we move it and psi_flags, which
currently only has 5 bits, close to the bitfield variable in_memstall
together. This way they will be cacheline aligned together.
* Patch 3 adds a prefetch logic before actually accessing the parent
cgroups, since the parent cgroups will always be accessed in the
following step.
* Patch 4 only calls record_times when the state actually changes to
save some uncessary accesses.
* Patch 5 adds psi_group for the root cgroup to remove the uncessary
if condition.
* Patch 6 uses printk_deferred_once to replace the psi_bug variable
and moves tasks[NR_RUNNING] which is most likely to happen ahead
in the if condition.
Thanks for reading. Comments and suggestions are very welcome!
Signed-off-by: Luka Bai <lukabai@tencent.com>
---
Luka Bai (6):
psi: move curr_in_memstall out of psi_group_change
psi: reorganize the psi members for cacheline benifits
psi: use prefetch to preread the parent groupc
psi: do not call record_times when the state is not changed
psi: add psi group for the root cgroup
psi: remove psi_bug and moves checking of NR_RUNNING ahead.
include/linux/psi.h | 2 +-
include/linux/psi_types.h | 20 +------------
include/linux/sched.h | 29 ++++++++++++++++---
kernel/cgroup/cgroup.c | 3 ++
kernel/fork.c | 10 +++++++
kernel/sched/psi.c | 71 ++++++++++++++++++++++++++++++-----------------
6 files changed, 85 insertions(+), 50 deletions(-)
---
base-commit: 972c53e0ec3abfc6f5fe2cb503640710fb23cf95
change-id: 20260512-psi_impr-f543a199f39d
Best regards,
--
Luka Bai <lukabai@tencent.com>
^ permalink raw reply
* [PATCH 1/6] psi: move curr_in_memstall out of psi_group_change
From: Luka Bai @ 2026-05-12 6:19 UTC (permalink / raw)
To: linux-mm
Cc: Johannes Weiner, Suren Baghdasaryan, Peter Ziljstra, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak,
Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport, Michal Hocko,
Kees Cook, Tejun Heo, Michal Koutný, linux-kernel, cgroups,
Luka Bai
In-Reply-To: <20260512-psi_impr-v1-0-2b7f10fdfad5@tencent.com>
From: Luka Bai <lukabai@tencent.com>
Variable curr_in_memstall is currently judged by accessing the
in_memstall of cpu_curr(cpu), which contains multiple times of
memory accessing. And it is now located in psi_group_change()
that will be called for each parent cgroup and it is redundant
sometimes since its value will not change for all these parent
cgroups.
So we move the variable outside for two reasons:
1. We save the extra calling for each parent cgroup so we avoid
these possible uncessary cacheline stall.
2. For function like psi_task_switch, we don't need to call the
cpu_curr(cpu) to get the task that is currently running in
the cpu runqueue. Under that context, "next" is absolutely the
running task so we can save some costly calling.
Signed-off-by: Luka Bai <lukabai@tencent.com>
---
kernel/sched/psi.c | 23 ++++++++++++++++-------
1 file changed, 16 insertions(+), 7 deletions(-)
diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index d9c9d9480a45..27097cb0dc79 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -795,7 +795,7 @@ static void record_times(struct psi_group_cpu *groupc, u64 now)
static void psi_group_change(struct psi_group *group, int cpu,
unsigned int clear, unsigned int set,
- u64 now, bool wake_clock)
+ u64 now, bool wake_clock, bool curr_in_memstall)
{
struct psi_group_cpu *groupc;
unsigned int t, m;
@@ -868,7 +868,7 @@ static void psi_group_change(struct psi_group *group, int cpu,
* task in a cgroup is in_memstall, the corresponding groupc
* on that cpu is in PSI_MEM_FULL state.
*/
- if (unlikely((state_mask & PSI_ONCPU) && cpu_curr(cpu)->in_memstall))
+ if (unlikely((state_mask & PSI_ONCPU) && curr_in_memstall))
state_mask |= (1 << PSI_MEM_FULL);
record_times(groupc, now);
@@ -910,6 +910,7 @@ void psi_task_change(struct task_struct *task, int clear, int set)
{
int cpu = task_cpu(task);
u64 now;
+ bool curr_in_memstall;
if (!task->pid)
return;
@@ -917,9 +918,11 @@ void psi_task_change(struct task_struct *task, int clear, int set)
psi_flags_change(task, clear, set);
psi_write_begin(cpu);
+ curr_in_memstall = cpu_curr(cpu)->in_memstall;
now = cpu_clock(cpu);
for_each_group(group, task_psi_group(task))
- psi_group_change(group, cpu, clear, set, now, true);
+ psi_group_change(group, cpu, clear, set, now, true,
+ curr_in_memstall);
psi_write_end(cpu);
}
@@ -929,11 +932,13 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next,
struct psi_group *common = NULL;
int cpu = task_cpu(prev);
u64 now;
+ bool curr_in_memstall = false;
psi_write_begin(cpu);
now = cpu_clock(cpu);
if (next->pid) {
+ curr_in_memstall = next->in_memstall;
psi_flags_change(next, 0, TSK_ONCPU);
/*
* Set TSK_ONCPU on @next's cgroups. If @next shares any
@@ -947,7 +952,8 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next,
common = group;
break;
}
- psi_group_change(group, cpu, 0, TSK_ONCPU, now, true);
+ psi_group_change(group, cpu, 0, TSK_ONCPU, now, true,
+ curr_in_memstall);
}
}
@@ -984,7 +990,8 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next,
for_each_group(group, task_psi_group(prev)) {
if (group == common)
break;
- psi_group_change(group, cpu, clear, set, now, wake_clock);
+ psi_group_change(group, cpu, clear, set, now, wake_clock,
+ curr_in_memstall);
}
/*
@@ -996,7 +1003,8 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next,
if ((prev->psi_flags ^ next->psi_flags) & ~TSK_ONCPU) {
clear &= ~TSK_ONCPU;
for_each_group(group, common)
- psi_group_change(group, cpu, clear, set, now, wake_clock);
+ psi_group_change(group, cpu, clear, set, now, wake_clock,
+ curr_in_memstall);
}
}
psi_write_end(cpu);
@@ -1236,7 +1244,8 @@ void psi_cgroup_restart(struct psi_group *group)
psi_write_begin(cpu);
now = cpu_clock(cpu);
- psi_group_change(group, cpu, 0, 0, now, true);
+ psi_group_change(group, cpu, 0, 0, now, true,
+ cpu_curr(cpu)->in_memstall);
psi_write_end(cpu);
}
}
--
2.52.0
^ permalink raw reply related
* [PATCH 2/6] psi: reorganize the psi members for cacheline benifits
From: Luka Bai @ 2026-05-12 6:19 UTC (permalink / raw)
To: linux-mm
Cc: Johannes Weiner, Suren Baghdasaryan, Peter Ziljstra, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak,
Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport, Michal Hocko,
Kees Cook, Tejun Heo, Michal Koutný, linux-kernel, cgroups,
Luka Bai
In-Reply-To: <20260512-psi_impr-v1-0-2b7f10fdfad5@tencent.com>
From: Luka Bai <lukabai@tencent.com>
Currently, we check whether the task needs to do psi accounting by
reading task->pid, which is not cacheline aligned with other psi
variables like in_memstall. This can generate some cacheline stall
from what perf-record indicates. So we would like to merge these
variables together.
However, directly switching order of pid and restart_block may cause
other cacheline problem in other scenorios which is hard to recognize
clearly. So we added need_psi bitfield variable to indicate the same psi
thing and put it together with in_memstall. The value of need_psi will
not be changed ever since the task gets created so there is no problem
about synchronization. Also, adding one bit to the bitfield variable
of unsigned int will not enlarge the size of task_struct or change the
memory pattern of task_struct at all.
Also, we put psi_flags which only has 5 bits long together with
in_memstall and need_psi too to make them all cacheline optimized.
5 extra bits can also be stuffed into one single unsigned int so it
will also not enlarge the size of task_struct, but on the contrary,
it will shrink the task_struct since we eliminate the psi_flags that
was put there independently as a unsigned int.
We also add NR_TSK_ONCPU and NR_PSI_ALL_COUNTS into the psi_task_count
enum definition to make the semantics clearer, and move the definition
from linux/psi_types.h into linux/sched.h since we need those enums in
linux/sched.h. These two revisions do not make any actual funtional
difference to the code.
Signed-off-by: Luka Bai <lukabai@tencent.com>
---
include/linux/psi_types.h | 20 +-------------------
include/linux/sched.h | 29 +++++++++++++++++++++++++----
kernel/fork.c | 10 ++++++++++
kernel/sched/psi.c | 6 +++---
4 files changed, 39 insertions(+), 26 deletions(-)
diff --git a/include/linux/psi_types.h b/include/linux/psi_types.h
index dd10c22299ab..5639dcdd90af 100644
--- a/include/linux/psi_types.h
+++ b/include/linux/psi_types.h
@@ -10,24 +10,6 @@
#ifdef CONFIG_PSI
-/* Tracked task states */
-enum psi_task_count {
- NR_IOWAIT,
- NR_MEMSTALL,
- NR_RUNNING,
- /*
- * For IO and CPU stalls the presence of running/oncpu tasks
- * in the domain means a partial rather than a full stall.
- * For memory it's not so simple because of page reclaimers:
- * they are running/oncpu while representing a stall. To tell
- * whether a domain has productivity left or not, we need to
- * distinguish between regular running (i.e. productive)
- * threads and memstall ones.
- */
- NR_MEMSTALL_RUNNING,
- NR_PSI_TASK_COUNTS = 4,
-};
-
/* Task state bitmasks */
#define TSK_IOWAIT (1 << NR_IOWAIT)
#define TSK_MEMSTALL (1 << NR_MEMSTALL)
@@ -35,7 +17,7 @@ enum psi_task_count {
#define TSK_MEMSTALL_RUNNING (1 << NR_MEMSTALL_RUNNING)
/* Only one task can be scheduled, no corresponding task count */
-#define TSK_ONCPU (1 << NR_PSI_TASK_COUNTS)
+#define TSK_ONCPU (1 << NR_TSK_ONCPU)
/* Resources that workloads could be stalled on */
enum psi_res {
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 368c7b4d7cb5..34d7f80531e7 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -817,6 +817,28 @@ struct kmap_ctrl {
#endif
};
+#ifdef CONFIG_PSI
+/* Tracked task states */
+enum psi_task_count {
+ NR_IOWAIT,
+ NR_MEMSTALL,
+ NR_RUNNING,
+ /*
+ * For IO and CPU stalls the presence of running/oncpu tasks
+ * in the domain means a partial rather than a full stall.
+ * For memory it's not so simple because of page reclaimers:
+ * they are running/oncpu while representing a stall. To tell
+ * whether a domain has productivity left or not, we need to
+ * distinguish between regular running (i.e. productive)
+ * threads and memstall ones.
+ */
+ NR_MEMSTALL_RUNNING,
+ NR_PSI_TASK_COUNTS,
+ NR_TSK_ONCPU = NR_PSI_TASK_COUNTS,
+ NR_PSI_ALL_COUNTS,
+};
+#endif
+
struct task_struct {
#ifdef CONFIG_THREAD_INFO_IN_TASK
/*
@@ -1030,6 +1052,9 @@ struct task_struct {
#ifdef CONFIG_PSI
/* Stalled due to lack of memory */
unsigned in_memstall:1;
+ unsigned need_psi:1;
+ /* Pressure stall state */
+ unsigned psi_flags:NR_PSI_ALL_COUNTS;
#endif
#ifdef CONFIG_PAGE_OWNER
/* Used by page_owner=on to detect recursion in page tracking. */
@@ -1299,10 +1324,6 @@ struct task_struct {
kernel_siginfo_t *last_siginfo;
struct task_io_accounting ioac;
-#ifdef CONFIG_PSI
- /* Pressure stall state */
- unsigned int psi_flags;
-#endif
#ifdef CONFIG_TASK_XACCT
/* Accumulated RSS usage: */
u64 acct_rss_mem1;
diff --git a/kernel/fork.c b/kernel/fork.c
index 0d97fd71d7f6..20b47c876b27 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -2177,6 +2177,16 @@ __latent_entropy struct task_struct *copy_process(
#ifdef CONFIG_PSI
p->psi_flags = 0;
+ /*
+ * Only setup need_psi to 1 for non-idle tasks. We
+ * also need to reset need_psi of idle tasks to 0 since
+ * their values are copied from the init task whose
+ * need_psi is not 0.
+ */
+ if (pid != &init_struct_pid)
+ p->need_psi = 1;
+ else
+ p->need_psi = 0;
#endif
task_io_accounting_init(&p->ioac);
diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index 27097cb0dc79..7374c05a5751 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -912,7 +912,7 @@ void psi_task_change(struct task_struct *task, int clear, int set)
u64 now;
bool curr_in_memstall;
- if (!task->pid)
+ if (!task->need_psi)
return;
psi_flags_change(task, clear, set);
@@ -937,7 +937,7 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next,
psi_write_begin(cpu);
now = cpu_clock(cpu);
- if (next->pid) {
+ if (next->need_psi) {
curr_in_memstall = next->in_memstall;
psi_flags_change(next, 0, TSK_ONCPU);
/*
@@ -957,7 +957,7 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next,
}
}
- if (prev->pid) {
+ if (prev->need_psi) {
int clear = TSK_ONCPU, set = 0;
bool wake_clock = true;
--
2.52.0
^ permalink raw reply related
* [PATCH 3/6] psi: use prefetch to preread the parent groupc
From: Luka Bai @ 2026-05-12 6:19 UTC (permalink / raw)
To: linux-mm
Cc: Johannes Weiner, Suren Baghdasaryan, Peter Ziljstra, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak,
Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport, Michal Hocko,
Kees Cook, Tejun Heo, Michal Koutný, linux-kernel, cgroups,
Luka Bai
In-Reply-To: <20260512-psi_impr-v1-0-2b7f10fdfad5@tencent.com>
From: Luka Bai <lukabai@tencent.com>
When doing psi_group_change, we always iterate all the cgroups from
the child all the way up to the root cgroup. They are all double link
list connected so it's hard for the CPU to prefetch this parent. So
we tried to add a prefetch for the parent groupc, and it has quite some
benefits for the final result.
Signed-off-by: Luka Bai <lukabai@tencent.com>
---
kernel/sched/psi.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index 7374c05a5751..9b7a85d1bc28 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -793,6 +793,15 @@ static void record_times(struct psi_group_cpu *groupc, u64 now)
#define for_each_group(iter, group) \
for (typeof(group) iter = group; iter; iter = iter->parent)
+static inline struct psi_group_cpu *prefetch_and_get_groupc(struct psi_group *group, int cpu)
+{
+ struct psi_group_cpu *groupc = per_cpu_ptr(group->pcpu, cpu);
+
+ if (group->parent)
+ prefetchw(per_cpu_ptr(group->parent->pcpu, cpu));
+ return groupc;
+}
+
static void psi_group_change(struct psi_group *group, int cpu,
unsigned int clear, unsigned int set,
u64 now, bool wake_clock, bool curr_in_memstall)
@@ -802,7 +811,7 @@ static void psi_group_change(struct psi_group *group, int cpu,
u32 state_mask;
lockdep_assert_rq_held(cpu_rq(cpu));
- groupc = per_cpu_ptr(group->pcpu, cpu);
+ groupc = prefetch_and_get_groupc(group, cpu);
/*
* Start with TSK_ONCPU, which doesn't have a corresponding
--
2.52.0
^ permalink raw reply related
* [PATCH 4/6] psi: do not call record_times when the state is not changed
From: Luka Bai @ 2026-05-12 6:20 UTC (permalink / raw)
To: linux-mm
Cc: Johannes Weiner, Suren Baghdasaryan, Peter Ziljstra, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak,
Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport, Michal Hocko,
Kees Cook, Tejun Heo, Michal Koutný, linux-kernel, cgroups,
Luka Bai
In-Reply-To: <20260512-psi_impr-v1-0-2b7f10fdfad5@tencent.com>
From: Luka Bai <lukabai@tencent.com>
In psi_group_change, record_times is always called no matter whether
the state_mask changes. Since it can cost some performance, we
choose to not to do it unconditionally. If the state has not changed,
we can keep the psi time unchanged.
This will not make any difference to the final result since when
we need to acquire the psi time, get_recent_times() will always
calculate the remaining time into the final result.
Signed-off-by: Luka Bai <lukabai@tencent.com>
---
kernel/sched/psi.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index 9b7a85d1bc28..4c4bd134c785 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -880,9 +880,15 @@ static void psi_group_change(struct psi_group *group, int cpu,
if (unlikely((state_mask & PSI_ONCPU) && curr_in_memstall))
state_mask |= (1 << PSI_MEM_FULL);
- record_times(groupc, now);
-
- groupc->state_mask = state_mask;
+ /*
+ * We only need to record times when the state changes. Or
+ * we can keep it unchanged and wait for get_recent_times()
+ * to handle the remaining time.
+ */
+ if (state_mask != groupc->state_mask) {
+ record_times(groupc, now);
+ groupc->state_mask = state_mask;
+ }
if (state_mask & group->rtpoll_states)
psi_schedule_rtpoll_work(group, 1, false);
--
2.52.0
^ permalink raw reply related
* [PATCH 5/6] psi: add psi group for the root cgroup
From: Luka Bai @ 2026-05-12 6:20 UTC (permalink / raw)
To: linux-mm
Cc: Johannes Weiner, Suren Baghdasaryan, Peter Ziljstra, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak,
Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport, Michal Hocko,
Kees Cook, Tejun Heo, Michal Koutný, linux-kernel, cgroups,
Luka Bai
In-Reply-To: <20260512-psi_impr-v1-0-2b7f10fdfad5@tencent.com>
From: Luka Bai <lukabai@tencent.com>
Cgroup_psi() now includes a condition, and checks against whether
the cgroup is the root cgroup to decide whether to use psi_system
instead of cgrp->psi. This is mostly because the default hierarchy
does not have any psi group attached. So we make psi_system as
its psi group, and remove the if condition in cgroup_psi().
Signed-off-by: Luka Bai <lukabai@tencent.com>
---
include/linux/psi.h | 2 +-
kernel/cgroup/cgroup.c | 3 +++
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/include/linux/psi.h b/include/linux/psi.h
index e0745873e3f2..8f2db511d051 100644
--- a/include/linux/psi.h
+++ b/include/linux/psi.h
@@ -34,7 +34,7 @@ __poll_t psi_trigger_poll(void **trigger_ptr, struct file *file,
#ifdef CONFIG_CGROUPS
static inline struct psi_group *cgroup_psi(struct cgroup *cgrp)
{
- return cgroup_ino(cgrp) == 1 ? &psi_system : cgrp->psi;
+ return cgrp->psi;
}
int psi_cgroup_alloc(struct cgroup *cgrp);
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index 43adc96c7f1a..357c68662d18 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -178,6 +178,9 @@ static DEFINE_PER_CPU(struct cgroup_rstat_base_cpu, root_rstat_base_cpu);
/* the default hierarchy */
struct cgroup_root cgrp_dfl_root = {
.cgrp.self.rstat_cpu = &root_rstat_cpu,
+#ifdef CONFIG_PSI
+ .cgrp.psi = &psi_system,
+#endif
.cgrp.rstat_base_cpu = &root_rstat_base_cpu,
};
EXPORT_SYMBOL_GPL(cgrp_dfl_root);
--
2.52.0
^ permalink raw reply related
* [PATCH 6/6] psi: remove psi_bug and moves checking of NR_RUNNING ahead.
From: Luka Bai @ 2026-05-12 6:20 UTC (permalink / raw)
To: linux-mm
Cc: Johannes Weiner, Suren Baghdasaryan, Peter Ziljstra, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak,
Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport, Michal Hocko,
Kees Cook, Tejun Heo, Michal Koutný, linux-kernel, cgroups,
Luka Bai
In-Reply-To: <20260512-psi_impr-v1-0-2b7f10fdfad5@tencent.com>
From: Luka Bai <lukabai@tencent.com>
During the accounting of psi states we'd like to do some bug detection
to make it more maintainable. And we use the variable psi_bug to make
it print once. We would like to use printk_deferred_once to replace the
usage of psi_bug since their effect are similar, and this can also
increase the readability.
Also, use likely and unlikely in these bug detection branches.
Signed-off-by: Luka Bai <lukabai@tencent.com>
---
kernel/sched/psi.c | 21 ++++++++-------------
1 file changed, 8 insertions(+), 13 deletions(-)
diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index 4c4bd134c785..70dd642af5e0 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -141,8 +141,6 @@
#include <linux/psi.h>
#include "sched.h"
-static int psi_bug __read_mostly;
-
DEFINE_STATIC_KEY_FALSE(psi_disabled);
static DEFINE_STATIC_KEY_TRUE(psi_cgroups_enabled);
@@ -262,7 +260,7 @@ static u32 test_states(unsigned int *tasks, u32 state_mask)
if (tasks[NR_RUNNING] && !oncpu)
state_mask |= BIT(PSI_CPU_FULL);
- if (tasks[NR_IOWAIT] || tasks[NR_MEMSTALL] || tasks[NR_RUNNING])
+ if (tasks[NR_RUNNING] || tasks[NR_MEMSTALL] || tasks[NR_IOWAIT])
state_mask |= BIT(PSI_NONIDLE);
return state_mask;
@@ -836,14 +834,13 @@ static void psi_group_change(struct psi_group *group, int cpu,
for (t = 0, m = clear; m; m &= ~(1 << t), t++) {
if (!(m & (1 << t)))
continue;
- if (groupc->tasks[t]) {
+ if (likely(groupc->tasks[t])) {
groupc->tasks[t]--;
- } else if (!psi_bug) {
- printk_deferred(KERN_ERR "psi: task underflow! cpu=%d t=%d tasks=[%u %u %u %u] clear=%x set=%x\n",
+ } else {
+ printk_deferred_once("psi: task underflow! cpu=%d t=%d tasks=[%u %u %u %u] clear=%x set=%x\n",
cpu, t, groupc->tasks[0],
groupc->tasks[1], groupc->tasks[2],
groupc->tasks[3], clear, set);
- psi_bug = 1;
}
}
@@ -908,13 +905,11 @@ static inline struct psi_group *task_psi_group(struct task_struct *task)
static void psi_flags_change(struct task_struct *task, int clear, int set)
{
- if (((task->psi_flags & set) ||
- (task->psi_flags & clear) != clear) &&
- !psi_bug) {
- printk_deferred(KERN_ERR "psi: inconsistent task state! task=%d:%s cpu=%d psi_flags=%x clear=%x set=%x\n",
+ if (unlikely(((task->psi_flags & set) ||
+ (task->psi_flags & clear) != clear))) {
+ printk_deferred_once("psi: inconsistent task state! task=%d:%s cpu=%d psi_flags=%x clear=%x set=%x\n",
task->pid, task->comm, task_cpu(task),
task->psi_flags, clear, set);
- psi_bug = 1;
}
task->psi_flags &= ~clear;
@@ -927,7 +922,7 @@ void psi_task_change(struct task_struct *task, int clear, int set)
u64 now;
bool curr_in_memstall;
- if (!task->need_psi)
+ if (unlikely(!task->need_psi))
return;
psi_flags_change(task, clear, set);
--
2.52.0
^ permalink raw reply related
* Re: [PATCH] cgroup/dmem: return -ENOMEM on failed pool preallocation
From: Guopeng Zhang @ 2026-05-12 7:04 UTC (permalink / raw)
To: Michal Koutný
Cc: Maarten Lankhorst, Maxime Ripard, Natalie Vock, Tejun Heo,
Johannes Weiner, cgroups, dri-devel, linux-kernel
In-Reply-To: <1778555389652035.138.seg@mailgw.kylinos.cn>
在 2026/5/11 21:03, Michal Koutný 写道:
> On Mon, May 11, 2026 at 09:31:50AM +0800, Guopeng Zhang <zhangguopeng@kylinos.cn> wrote:
>> get_cg_pool_unlocked() handles allocation failures under dmemcg_lock by
>> dropping the lock, preallocating a pool with GFP_KERNEL, and retrying the
>> locked lookup and creation path.
>>
>> If the fallback allocation fails too, pool remains NULL. Since the loop
>> condition is while (!pool), the function can keep retrying instead of
>> propagating the allocation failure to the caller.
>
> This implies that it's OK when the function keeps retrying with
> allocpool != NULL (and repeated WARN_ON()s)?
Hi Michal,
Thanks for taking a look.
No, that was not what I meant to imply. The commit message was not precise
enough there.
The intended normal retry is only for the case where the GFP_NOWAIT
allocation under dmemcg_lock fails. In that case, get_cg_pool_unlocked()
drops the lock, preallocates one pool with GFP_KERNEL, and the next locked
retry consumes that preallocated pool and clears allocpool.
So allocpool != NULL together with another -ENOMEM return is not expected to
be a normal retry path. The WARN_ON(allocpool) branch looks defensive, and I
agree that repeatedly continuing from there would not be useful if it ever
fired.
>> Set pool to ERR_PTR(-ENOMEM) when the fallback allocation fails so the
>> loop exits through the existing common return path. The callers already
>> handle ERR_PTR() from get_cg_pool_unlocked(), so this restores the
>> expected error path.
>
> If the callers can handle it, maybe there's no need to retry at all.
> Perhpas dmem fellows can step in here.My understanding is that the retry still has a purpose independent of the
callers' ability to handle ERR_PTR().
The first allocation attempt happens in alloc_pool_single() while
dmemcg_lock is held, so it uses GFP_NOWAIT. If that fails,
get_cg_pool_unlocked() drops the lock and preallocates one pool with the
default GFP_KERNEL context. The next locked retry then consumes that
preallocated pool instead of trying another GFP_NOWAIT allocation for that
pool.
So callers can handle the final ERR_PTR() result, but the fallback
preallocation gives the allocation a chance to succeed in a less
constrained context before reporting -ENOMEM. That said, whether this
retry policy is desirable is a dmem design question, so input from dmem
folks would be helpful.
>>
>> Fixes: b168ed458dde ("kernel/cgroup: Add "dmem" memory accounting cgroup")
>> Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
>> ---
>> kernel/cgroup/dmem.c | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
>> index 1ab1fb47f271..4753a67d0f0f 100644
>> --- a/kernel/cgroup/dmem.c
>> +++ b/kernel/cgroup/dmem.c
>> @@ -602,6 +602,7 @@ get_cg_pool_unlocked(struct dmemcg_state *cg, struct dmem_cgroup_region *region)
>> pool = NULL;
>
> This 2nd pool zeroing seems pointless.
Agreed.
Since Tejun has already applied the fix, I will wait for the discussion
before sending any follow-up. If we keep the current retry scheme, a small
cleanup can make this path clearer.
Thanks,
Guopeng
^ permalink raw reply
* [PATCH v2] selftests/cgroup: check malloc return value in alloc_anon functions
From: Hongfu Li @ 2026-05-12 7:14 UTC (permalink / raw)
To: hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song, tj,
mkoutny, shuah, vishal.moola
Cc: cgroups, linux-mm, linux-kselftest, linux-kernel, Hongfu Li
The alloc_anon() function calls malloc() without checking for a NULL
return. If memory allocation fails, a NULL pointer dereference will
occur when accessing the buffer.
Add proper error handling to return -1 when malloc() fails in all
four alloc_anon variants:
- alloc_anon()
- alloc_anon_50M_check()
- alloc_anon_noexit()
- alloc_anon_50M_check_swap()
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Reviewed-by: Vishal Moola <vishal.moola@gmail.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
---
v2:
- Refactored repeated malloc() patterns in alloc_anon_* into a helper
function.
---
.../selftests/cgroup/test_memcontrol.c | 53 ++++++++++---------
1 file changed, 27 insertions(+), 26 deletions(-)
diff --git a/tools/testing/selftests/cgroup/test_memcontrol.c b/tools/testing/selftests/cgroup/test_memcontrol.c
index b43da9bc20c4..21aedb35cc12 100644
--- a/tools/testing/selftests/cgroup/test_memcontrol.c
+++ b/tools/testing/selftests/cgroup/test_memcontrol.c
@@ -55,15 +55,31 @@ int alloc_pagecache(int fd, size_t size)
return -1;
}
-int alloc_anon(const char *cgroup, void *arg)
+static char *alloc_and_populate_anon(size_t size)
{
- size_t size = (unsigned long)arg;
char *buf, *ptr;
buf = malloc(size);
+ if (buf == NULL) {
+ fprintf(stderr, "malloc() failed\n");
+ return NULL;
+ }
+
for (ptr = buf; ptr < buf + size; ptr += PAGE_SIZE)
*ptr = 0;
+ return buf;
+}
+
+int alloc_anon(const char *cgroup, void *arg)
+{
+ size_t size = (unsigned long)arg;
+ char *buf;
+
+ buf = alloc_and_populate_anon(size);
+ if (!buf)
+ return -1;
+
free(buf);
return 0;
}
@@ -174,18 +190,13 @@ static int test_memcg_subtree_control(const char *root)
static int alloc_anon_50M_check(const char *cgroup, void *arg)
{
size_t size = MB(50);
- char *buf, *ptr;
+ char *buf;
long anon, current;
int ret = -1;
- buf = malloc(size);
- if (buf == NULL) {
- fprintf(stderr, "malloc() failed\n");
+ buf = alloc_and_populate_anon(size);
+ if (!buf)
return -1;
- }
-
- for (ptr = buf; ptr < buf + size; ptr += PAGE_SIZE)
- *ptr = 0;
current = cg_read_long(cgroup, "memory.current");
if (current < size)
@@ -406,16 +417,11 @@ static int alloc_anon_noexit(const char *cgroup, void *arg)
{
int ppid = getppid();
size_t size = (unsigned long)arg;
- char *buf, *ptr;
+ char *buf;
- buf = malloc(size);
- if (buf == NULL) {
- fprintf(stderr, "malloc() failed\n");
+ buf = alloc_and_populate_anon(size);
+ if (!buf)
return -1;
- }
-
- for (ptr = buf; ptr < buf + size; ptr += PAGE_SIZE)
- *ptr = 0;
while (getppid() == ppid)
sleep(1);
@@ -990,18 +996,13 @@ static int alloc_anon_50M_check_swap(const char *cgroup, void *arg)
{
long mem_max = (long)arg;
size_t size = MB(50);
- char *buf, *ptr;
+ char *buf;
long mem_current, swap_current;
int ret = -1;
- buf = malloc(size);
- if (buf == NULL) {
- fprintf(stderr, "malloc() failed\n");
+ buf = alloc_and_populate_anon(size);
+ if (!buf)
return -1;
- }
-
- for (ptr = buf; ptr < buf + size; ptr += PAGE_SIZE)
- *ptr = 0;
mem_current = cg_read_long(cgroup, "memory.current");
if (!mem_current || !values_close(mem_current, mem_max, 3))
--
2.25.1
^ permalink raw reply related
* Re: [PATCH v2 00/10] sched: Flatten the pick
From: Peter Zijlstra @ 2026-05-12 8:10 UTC (permalink / raw)
To: Tejun Heo
Cc: mingo, longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <agIswZpCxlsQ2Xdk@slm.duckdns.org>
On Mon, May 11, 2026 at 09:23:45AM -1000, Tejun Heo wrote:
> Hello, Peter.
>
> On Mon, May 11, 2026 at 01:31:04PM +0200, Peter Zijlstra wrote:
> > So cgroup scheduling has always been a pain in the arse. The problems start
> > with weight distribution and end with hierachical picks and it all sucks.
> >
> > The problems with weight distribution are related to that infernal global
> > fraction:
> >
> > tg->w * grq_i->w
> > ge_i->w = ----------------
> > \Sum_j grq_j->w
> >
> > which we've approximated reasonably well by now. However, the immediate
> > consequence of this fraction is that the total group weight (tg->w) gets
> > fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup
> > weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine
> > with the fact that 256 CPU systems are relatively common these days, this
> > becomes painful.
> >
> > The common 'solution' is to inflate the group weight by 'nr_cpus'; the
> > immediate problem with that is that when all load of a group gets concentrated
> > on a single CPU, the per-cpu cgroup weight becomes insanely large, easily
> > exceeding nice -20.
> >
> > Additionally there are numerical limits on the max weight you can have before
> > the math starts suffering overflows. As such there is a definite limit on the
> > total group weight. Which has annoyed people ;-)
> >
> > The first few patches add a knob /debug/sched/cgroup_mode and a few different
> > options on how to deal with this. My favourite is 'concur', but obviously that
> > is also the most expensive one :-/ It adds a tg->tasks counter which makes the
> > update_tg_load_avg() thing more expensive.
>
> Ignoring fixed math accuracy problems, isn't the root problem here that
> every thread in the root cgroup competes as if each is its own cgroup? ie.
> Isn't the canonical solution here to create an enveloping group, at least
> for share calculation purposes, for root threads and then assign them some
> weight so that they compete in the same way that other cgroups do? Then, the
> different modes go away or rather whatever the user wants can be expressed
> via root's weight if that's to be made configurable.
As long as the total group weight is a fraction; and it sorta has to be.
You can run into trouble by stacking that fraction.
Take 256 CPUs and a group weight of 1024. Then each CPU gets a weight of
1/256 or 4. Even if we increase the internal accuracy to 20 bits (we do
on 64bit) then this becomes 4096, do this for 2 more levels in the
hierarchy and you're down to scraping the barrel again.
So if each level runs at a fraction f of the level above, then level n
runs at f^n. Moving root into a phantom group at level 1, only solves
the problem against other tasks at level 1, but then you have the same
problem again at level 2 and below.
Both the numerical problems and the scale problem of the root group can
be avoided if we can get the average/nominal fraction to be near 1.
The 'normal' way around this is to ensure the group weight is nr_cpus *
1024, then, when everybody is running, the per CPU weight is 1024 or 1
and the continued fraction is also 1-ish. This is why people like to
increase the max group weight.
Trouble is of course that if not all CPUs are busy, with the extreme
being only a single CPU carrying that weight of nr_cpus*1024, this then
causes trouble because that one CPU gets overloaded.
One of the options is to simply put a max on the single CPU load; which
is the crudest option to just make it 'work'. The one I favour though is
the one where we scale the group weight by: 'min(cpumas, nr_tasks)'.
Anyway, this is why I've been looking at these alternative weight
schemes, to get the nominal fraction near 1 and make these problems go
away. It is both the numerical issues and the disparity between levels
(with root being at level 0 being the most obvious).
Does that make sense?
^ permalink raw reply
* [PATCH v4 0/5] Add reclaim to the dmem cgroup controller
From: Thomas Hellström @ 2026-05-12 8:24 UTC (permalink / raw)
To: intel-xe
Cc: Thomas Hellström, Natalie Vock, Johannes Weiner, Tejun Heo,
Michal Koutný, cgroups, Huang Rui, Matthew Brost,
Matthew Auld, Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Simona Vetter, David Airlie, Christian König, Alex Deucher,
Rodrigo Vivi, dri-devel, amd-gfx, linux-kernel
When writing a "max" limit lower than the current usage, the
existing code silently failed. This series aims to improve
on that by returning -EBUSY on failure and also attempt
to synchronously reclaim device memory to push the usage
under the new max limit to avoid the error.
Patch 1 fixes a pre-existing amdgpu_vram_mgr_init() error path
Patch 2 implements and documents a reclaim callback interface
for the dmem controller.
Patch 3 implements a TTM reclaim callback.
Patch 4-5 hooks up the reclaim callback to the dmem cgroups-
aware drivers xe and amdgpu.
v2:
- Remove the error propagation that was in a previous series (Maarten)
- A number of updates in patch 1. See its commit message for
details (Maarten)
v3:
- Add patch 1 fixing a pre-existing amdgpu_vram_mgr_init() error path
bug where drmm_cgroup_register_region() was called before
INIT_LIST_HEAD() and gpu_buddy_init(), causing a kernel panic on
failure. (Sashiko-bot)
- Use an rwsem to protect reclaim callback registration and region
unregister against concurrent reclaim invocations. (Sashiko-bot)
- Fix ttm_resource_manager_set_dmem_region() storing an error pointer
in man->cg unconditionally. (Sashiko-bot)
- Fix kernel-doc function name format for ttm_bo_evict_cgroup() and
ttm_resource_manager_set_dmem_region().
v4:
- Rebased on drm-tip; dropped the XE_PL_STOLEN guard in the xe patch
as stolen memory uses a separate TTM manager.
User-space tests are at
https://patchwork.freedesktop.org/series/163935/
Test-with: 20260428065411.4222-1-thomas.hellstrom@linux.intel.com
Thomas Hellström (5):
drm/amdgpu: Fix init ordering in amdgpu_vram_mgr_init()
cgroup/dmem: Add reclaim callback for lowering max below current usage
drm/ttm: Hook up a cgroup-aware reclaim callback for the dmem
controller
drm/xe: Wire up dmem cgroup reclaim for VRAM manager
drm/amdgpu: Wire up dmem cgroup reclaim for VRAM manager
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 10 +-
drivers/gpu/drm/ttm/ttm_bo.c | 95 ++++++++++++++++-
drivers/gpu/drm/ttm/ttm_bo_util.c | 3 +-
drivers/gpu/drm/ttm/ttm_resource.c | 37 +++++++
drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 14 ++-
include/drm/ttm/ttm_bo.h | 10 ++
include/drm/ttm/ttm_resource.h | 4 +
include/linux/cgroup_dmem.h | 24 +++++
kernel/cgroup/dmem.c | 106 +++++++++++++++++--
10 files changed, 283 insertions(+), 22 deletions(-)
--
2.54.0
^ permalink raw reply
* [PATCH v4 1/5] drm/amdgpu: Fix init ordering in amdgpu_vram_mgr_init()
From: Thomas Hellström @ 2026-05-12 8:24 UTC (permalink / raw)
To: intel-xe
Cc: Thomas Hellström, Sashiko-bot, Friedrich Vock,
Maarten Lankhorst, Tejun Heo, Maxime Ripard, Christian König,
Alex Deucher, amd-gfx, dri-devel, stable, Natalie Vock,
Johannes Weiner, Michal Koutný, cgroups, Huang Rui,
Matthew Brost, Matthew Auld, Maarten Lankhorst, Thomas Zimmermann,
Simona Vetter, David Airlie, Rodrigo Vivi, linux-kernel
In-Reply-To: <20260512082406.44470-1-thomas.hellstrom@linux.intel.com>
drmm_cgroup_register_region() is called before INIT_LIST_HEAD() and
gpu_buddy_init() in amdgpu_vram_mgr_init(). If it fails, the function
returns early and bypasses those initializations.
Since adev->mman.initialized is set to true before amdgpu_vram_mgr_init()
is called, a failure triggers amdgpu_ttm_fini(), which calls
amdgpu_vram_mgr_fini(), which then:
- Calls list_for_each_entry_safe() on reservations_pending and
reserved_pages, whose list_head::next pointers are zero-initialized
(NULL). The loop does not recognize them as empty and dereferences NULL.
- Calls gpu_buddy_fini(), which iterates free_trees[] unconditionally
via for_each_free_tree(). Since mm->free_trees is NULL
(never allocated), this dereferences NULL.
Both result in a kernel panic on the module load error path.
Fix by moving drmm_cgroup_register_region() to after the list and buddy
allocator are fully initialized, so the teardown path is safe to run.
Reported-by: Sashiko-bot <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260428073116.15687-1-thomas.hellstrom@linux.intel.com?part=4
Fixes: 2b624a2c1865 ("drm/ttm: Handle cgroup based eviction in TTM")
Cc: Friedrich Vock <friedrich.vock@gmx.de>
Cc: Maarten Lankhorst <dev@lankhorst.se>
Cc: Tejun Heo <tj@kernel.org>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Christian König <christian.koenig@amd.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: <stable@vger.kernel.org> # v6.14+
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2a241a5b12c4..ac3f71d77140 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
@@ -918,9 +918,6 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev)
struct ttm_resource_manager *man = &mgr->manager;
int err;
- man->cg = drmm_cgroup_register_region(adev_to_drm(adev), "vram", adev->gmc.real_vram_size);
- if (IS_ERR(man->cg))
- return PTR_ERR(man->cg);
ttm_resource_manager_init(man, &adev->mman.bdev,
adev->gmc.real_vram_size);
@@ -935,6 +932,10 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev)
if (err)
return err;
+ man->cg = drmm_cgroup_register_region(adev_to_drm(adev), "vram", adev->gmc.real_vram_size);
+ if (IS_ERR(man->cg))
+ return PTR_ERR(man->cg);
+
ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_VRAM, &mgr->manager);
ttm_resource_manager_set_used(man, true);
return 0;
--
2.54.0
^ permalink raw reply related
* [PATCH v4 2/5] cgroup/dmem: Add reclaim callback for lowering max below current usage
From: Thomas Hellström @ 2026-05-12 8:24 UTC (permalink / raw)
To: intel-xe
Cc: Thomas Hellström, Natalie Vock, Johannes Weiner, Tejun Heo,
Michal Koutný, cgroups, Huang Rui, Matthew Brost,
Matthew Auld, Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Simona Vetter, David Airlie, Christian König, Alex Deucher,
Rodrigo Vivi, dri-devel, amd-gfx, linux-kernel
In-Reply-To: <20260512082406.44470-1-thomas.hellstrom@linux.intel.com>
Add an optional reclaim callback to struct dmem_cgroup_region. When
dmem.max is set below the current usage of a cgroup pool, the new limit
is applied immediately (so that concurrent allocations are throttled
while reclaim is in progress) and then the driver is asked to evict
memory to bring usage back below the limit.
Reclaim is attempted up to a bounded number of times. No error is
returned to userspace if usage remains above the limit after reclaim,
and a pending signal will abort the reclaim loop early. This matches
the behavior of memory.max in the memory cgroup controller.
Also honor O_NONBLOCK so that if that flag is set during the
max value write, no reclaim is initiated. The idea is to avoid
charging the reclaim cost to the writer of the max value.
v2:
- Write max before reclaim is attempted (Maarten)
- Let signals abort the reclaim without error (Maarten)
- If a new max value is written with the O_NONBLOCK flag,
reclaim is not attempted (Maarten)
- Extract region from the pool parameter rather than
passing it explicitly to set_resource_xxx().
v3:
- Use an rwsem to protect reclaim callback registration and
region unregister against concurrent reclaim invocations,
ensuring reclaim_priv is visible when the callback is
invoked. (Sashiko-bot)
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
include/linux/cgroup_dmem.h | 24 ++++++++
kernel/cgroup/dmem.c | 106 +++++++++++++++++++++++++++++++++---
2 files changed, 121 insertions(+), 9 deletions(-)
diff --git a/include/linux/cgroup_dmem.h b/include/linux/cgroup_dmem.h
index dd4869f1d736..c3bce21cbe80 100644
--- a/include/linux/cgroup_dmem.h
+++ b/include/linux/cgroup_dmem.h
@@ -14,6 +14,21 @@ struct dmem_cgroup_pool_state;
/* Opaque definition of a cgroup region, used internally */
struct dmem_cgroup_region;
+/**
+ * typedef dmem_cgroup_reclaim_fn_t - Reclaim callback for a dmem cgroup region.
+ * @pool: The cgroup pool that needs memory reclaimed.
+ * @target_bytes: Minimum number of bytes the driver should attempt to free.
+ * @priv: Private data registered with dmem_cgroup_region_set_reclaim().
+ *
+ * Called by the dmem cgroup controller when dmem.max is set below the current
+ * usage of @pool. The driver should evict at least @target_bytes of memory
+ * from @pool. May be called multiple times if usage remains above the limit.
+ *
+ * Return: 0 if progress was made, negative error code otherwise.
+ */
+typedef int (*dmem_cgroup_reclaim_fn_t)(struct dmem_cgroup_pool_state *pool,
+ u64 target_bytes, void *priv);
+
#if IS_ENABLED(CONFIG_CGROUP_DMEM)
struct dmem_cgroup_region *dmem_cgroup_register_region(u64 size, const char *name_fmt, ...) __printf(2,3);
void dmem_cgroup_unregister_region(struct dmem_cgroup_region *region);
@@ -26,6 +41,9 @@ bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
bool ignore_low, bool *ret_hit_low);
void dmem_cgroup_pool_state_put(struct dmem_cgroup_pool_state *pool);
+void dmem_cgroup_region_set_reclaim(struct dmem_cgroup_region *region,
+ dmem_cgroup_reclaim_fn_t reclaim,
+ void *priv);
#else
static inline __printf(2,3) struct dmem_cgroup_region *
dmem_cgroup_register_region(u64 size, const char *name_fmt, ...)
@@ -62,5 +80,11 @@ bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
static inline void dmem_cgroup_pool_state_put(struct dmem_cgroup_pool_state *pool)
{ }
+static inline void
+dmem_cgroup_region_set_reclaim(struct dmem_cgroup_region *region,
+ dmem_cgroup_reclaim_fn_t reclaim,
+ void *priv)
+{ }
+
#endif
#endif /* _CGROUP_DMEM_H */
diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
index 1ab1fb47f271..5fd5a1634d21 100644
--- a/kernel/cgroup/dmem.c
+++ b/kernel/cgroup/dmem.c
@@ -51,6 +51,20 @@ struct dmem_cgroup_region {
* No new pools should be added to the region afterwards.
*/
bool unregistered;
+
+ /**
+ * @reclaim: Optional callback invoked when dmem.max is set below the
+ * current usage of a pool. The driver should attempt to free at least
+ * @target_bytes from @pool. May be called multiple times if usage
+ * remains above the limit after returning.
+ */
+ dmem_cgroup_reclaim_fn_t reclaim;
+
+ /** @reclaim_priv: Private data passed to @reclaim. */
+ void *reclaim_priv;
+
+ /** @unregister_sem: Protect @reclaim while it is running. */
+ struct rw_semaphore unregister_sem;
};
struct dmemcg_state {
@@ -145,21 +159,58 @@ static void free_cg_pool(struct dmem_cgroup_pool_state *pool)
}
static void
-set_resource_min(struct dmem_cgroup_pool_state *pool, u64 val)
+set_resource_min(struct dmem_cgroup_pool_state *pool, u64 val, bool nonblock)
{
page_counter_set_min(&pool->cnt, val);
}
static void
-set_resource_low(struct dmem_cgroup_pool_state *pool, u64 val)
+set_resource_low(struct dmem_cgroup_pool_state *pool, u64 val, bool nonblock)
{
page_counter_set_low(&pool->cnt, val);
}
static void
-set_resource_max(struct dmem_cgroup_pool_state *pool, u64 val)
+set_resource_max(struct dmem_cgroup_pool_state *pool, u64 val, bool nonblock)
{
- page_counter_set_max(&pool->cnt, val);
+ struct dmem_cgroup_region *region = pool->region;
+
+ /*
+ * Always update the limit, even if usage currently exceeds it.
+ * Concurrent allocations will be throttled against the new limit
+ * while reclaim is in progress.
+ */
+ xchg(&pool->cnt.max, (unsigned long)val);
+
+ if (nonblock || !READ_ONCE(region->reclaim))
+ return;
+
+ for (int retries = 5; retries > 0; retries--) {
+ u64 usage = page_counter_read(&pool->cnt);
+ int ret;
+
+ if (usage <= val)
+ break;
+
+ if (signal_pending(current))
+ break;
+
+ /* Block unregister until the reclaim callback completes. */
+ if (down_read_interruptible(®ion->unregister_sem))
+ break;
+
+ if (!region->reclaim) {
+ up_read(®ion->unregister_sem);
+ break;
+ }
+
+ ret = region->reclaim(pool, usage - val, region->reclaim_priv);
+ up_read(®ion->unregister_sem);
+ if (ret)
+ break;
+
+ cond_resched();
+ }
}
static u64 get_resource_low(struct dmem_cgroup_pool_state *pool)
@@ -184,9 +235,9 @@ static u64 get_resource_current(struct dmem_cgroup_pool_state *pool)
static void reset_all_resource_limits(struct dmem_cgroup_pool_state *rpool)
{
- set_resource_min(rpool, 0);
- set_resource_low(rpool, 0);
- set_resource_max(rpool, PAGE_COUNTER_MAX);
+ set_resource_min(rpool, 0, false);
+ set_resource_low(rpool, 0, false);
+ set_resource_max(rpool, PAGE_COUNTER_MAX, false);
}
static void dmemcs_offline(struct cgroup_subsys_state *css)
@@ -491,6 +542,12 @@ void dmem_cgroup_unregister_region(struct dmem_cgroup_region *region)
region->unregistered = true;
spin_unlock(&dmemcg_lock);
+ /* Ensure all reclaim() callbacks have finished. */
+ down_write(®ion->unregister_sem);
+ /* Pairs with READ_ONCE() in set_resource_max() */
+ WRITE_ONCE(region->reclaim, NULL);
+ up_write(®ion->unregister_sem);
+
kref_put(®ion->ref, dmemcg_free_region);
}
EXPORT_SYMBOL_GPL(dmem_cgroup_unregister_region);
@@ -530,6 +587,7 @@ struct dmem_cgroup_region *dmem_cgroup_register_region(u64 size, const char *fmt
INIT_LIST_HEAD(&ret->pools);
ret->name = region_name;
ret->size = size;
+ init_rwsem(&ret->unregister_sem);
kref_init(&ret->ref);
spin_lock(&dmemcg_lock);
@@ -568,6 +626,34 @@ void dmem_cgroup_pool_state_put(struct dmem_cgroup_pool_state *pool)
}
EXPORT_SYMBOL_GPL(dmem_cgroup_pool_state_put);
+/**
+ * dmem_cgroup_region_set_reclaim() - Register a reclaim callback on a region.
+ * @region: The region to register the callback for.
+ * @reclaim: Callback to invoke when dmem.max is set below current usage.
+ * Called with the pool that needs reclaiming and the number of
+ * bytes to free. Returns 0 on progress, negative on failure.
+ * @priv: Opaque pointer passed back to @reclaim.
+ *
+ * When dmem.max is lowered below the current usage of a cgroup pool, the
+ * dmem controller will call @reclaim with a target number of bytes to free.
+ * After @reclaim returns the controller retries setting the limit; if usage
+ * is still too high it calls @reclaim again, up to a bounded retry count.
+ */
+void dmem_cgroup_region_set_reclaim(struct dmem_cgroup_region *region,
+ dmem_cgroup_reclaim_fn_t reclaim,
+ void *priv)
+{
+ if (!region)
+ return;
+
+ down_write(®ion->unregister_sem);
+ region->reclaim_priv = priv;
+ /* Pairs with READ_ONCE() in set_resource_max() */
+ WRITE_ONCE(region->reclaim, reclaim);
+ up_write(®ion->unregister_sem);
+}
+EXPORT_SYMBOL_GPL(dmem_cgroup_region_set_reclaim);
+
static struct dmem_cgroup_pool_state *
get_cg_pool_unlocked(struct dmemcg_state *cg, struct dmem_cgroup_region *region)
{
@@ -725,9 +811,10 @@ static int dmemcg_parse_limit(char *options, u64 *new_limit)
static ssize_t dmemcg_limit_write(struct kernfs_open_file *of,
char *buf, size_t nbytes, loff_t off,
- void (*apply)(struct dmem_cgroup_pool_state *, u64))
+ void (*apply)(struct dmem_cgroup_pool_state *, u64, bool))
{
struct dmemcg_state *dmemcs = css_to_dmemcs(of_css(of));
+ bool nonblock = of->file->f_flags & O_NONBLOCK;
int err = 0;
while (buf && !err) {
@@ -772,7 +859,8 @@ static ssize_t dmemcg_limit_write(struct kernfs_open_file *of,
}
/* And commit */
- apply(pool, new_limit);
+ apply(pool, new_limit, nonblock);
+
dmemcg_pool_put(pool);
out_put:
--
2.54.0
^ permalink raw reply related
* [PATCH v4 3/5] drm/ttm: Hook up a cgroup-aware reclaim callback for the dmem controller
From: Thomas Hellström @ 2026-05-12 8:24 UTC (permalink / raw)
To: intel-xe
Cc: Thomas Hellström, Natalie Vock, Johannes Weiner, Tejun Heo,
Michal Koutný, cgroups, Huang Rui, Matthew Brost,
Matthew Auld, Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Simona Vetter, David Airlie, Christian König, Alex Deucher,
Rodrigo Vivi, dri-devel, amd-gfx, linux-kernel
In-Reply-To: <20260512082406.44470-1-thomas.hellstrom@linux.intel.com>
Add ttm_bo_evict_cgroup() to evict buffer objects charged to a specific
dmem cgroup pool from a resource manager's LRU until a byte target is
met. Add ttm_resource_manager_set_dmem_region() to register the TTM
eviction path as the reclaim callback for a dmem cgroup region.
The eviction context is interruptible; signals abort the operation and
propagate back through the write() syscall.
Introduce a new mode for the bo LRU walker so that sleeping locks
can be taken. This can be used when the caller doesn't hold any
previous dma_resv locks, and where it intends to hold at most
one lock at a time.
Like the rest of the TTM eviction this should sooner than later
be converted to full WW transactions.
v3:
- Fix ttm_resource_manager_set_dmem_region() storing an error pointer
in man->cg unconditionally. (Sashiko-bot)
- Fix kernel-doc function name format for ttm_bo_evict_cgroup() and
ttm_resource_manager_set_dmem_region().
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/ttm/ttm_bo.c | 95 +++++++++++++++++++++++++++++-
drivers/gpu/drm/ttm/ttm_bo_util.c | 3 +-
drivers/gpu/drm/ttm/ttm_resource.c | 37 ++++++++++++
include/drm/ttm/ttm_bo.h | 10 ++++
include/drm/ttm/ttm_resource.h | 4 ++
5 files changed, 145 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 293401705542..3d0872f2f14d 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -515,12 +515,20 @@ static s64 ttm_bo_evict_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *
{
struct ttm_bo_evict_walk *evict_walk =
container_of(walk, typeof(*evict_walk), walk);
+ /* Capture size before eviction in case res is cleared. */
+ s64 bo_size = bo->base.size;
s64 lret;
if (!dmem_cgroup_state_evict_valuable(evict_walk->limit_pool, bo->resource->css,
evict_walk->try_low, &evict_walk->hit_low))
return 0;
+ /*
+ * evict_walk->place is NULL in cgroup drain mode. Drivers'
+ * eviction_valuable() callbacks must handle a NULL place, treating it
+ * as "any placement": the TTM base implementation already does so via
+ * ttm_resource_intersects().
+ */
if (bo->pin_count || !bo->bdev->funcs->eviction_valuable(bo, evict_walk->place))
return 0;
@@ -536,11 +544,15 @@ static s64 ttm_bo_evict_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *
goto out;
evict_walk->evicted++;
- if (evict_walk->res)
+ if (evict_walk->res) {
lret = ttm_resource_alloc(evict_walk->evictor, evict_walk->place,
evict_walk->res, NULL);
- if (lret == 0)
- return 1;
+ if (lret == 0)
+ return 1;
+ } else {
+ /* Cgroup drain: return bytes freed for byte-denominated progress. */
+ return bo_size;
+ }
out:
/* Errors that should terminate the walk. */
if (lret == -ENOSPC)
@@ -614,6 +626,83 @@ static int ttm_bo_evict_alloc(struct ttm_device *bdev,
return 0;
}
+/**
+ * ttm_bo_evict_cgroup() - Evict buffer objects charged to a specific cgroup.
+ * @bdev: The TTM device.
+ * @man: The resource manager whose LRU to walk.
+ * @limit_pool: The cgroup pool state whose members should be evicted.
+ * @target_bytes: Number of bytes to free.
+ * @ctx: The TTM operation context.
+ *
+ * Walk the LRU of @man and evict buffer objects that are charged to the
+ * cgroup identified by @limit_pool, until at least @target_bytes have been
+ * freed. Mirrors the two-pass (trylock -> sleeping-lock, low-watermark)
+ * strategy used by ttm_bo_evict_alloc().
+ *
+ * Return: >= @target_bytes on full success, 0..target_bytes-1 if partial,
+ * negative error code on fatal error.
+ */
+s64 ttm_bo_evict_cgroup(struct ttm_device *bdev,
+ struct ttm_resource_manager *man,
+ struct dmem_cgroup_pool_state *limit_pool,
+ s64 target_bytes,
+ struct ttm_operation_ctx *ctx)
+{
+ struct ttm_bo_evict_walk evict_walk = {
+ .walk = {
+ .ops = &ttm_evict_walk_ops,
+ .arg = { .ctx = ctx },
+ },
+ .limit_pool = limit_pool,
+ /* place, evictor, res left NULL: selects cgroup drain mode */
+ };
+ s64 lret, pass;
+
+ evict_walk.walk.arg.trylock_only = true;
+ lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, target_bytes);
+ if (lret < 0 || lret >= target_bytes)
+ return lret;
+
+ /* Second pass: also evict BOs at the low watermark. */
+ if (evict_walk.hit_low) {
+ evict_walk.try_low = true;
+ pass = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man,
+ target_bytes - lret);
+ if (pass < 0)
+ return pass;
+ lret += pass;
+ if (lret >= target_bytes)
+ return lret;
+ }
+
+ /* Full sleeping-lock pass for remaining target. */
+ evict_walk.try_low = evict_walk.hit_low = false;
+ evict_walk.walk.arg.trylock_only = false;
+
+retry:
+ evict_walk.walk.arg.sleeping_lock = true;
+ do {
+ evict_walk.evicted = 0;
+ pass = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man,
+ target_bytes - lret);
+ if (pass < 0) {
+ lret = pass;
+ goto out;
+ }
+ lret += pass;
+ } while (lret < target_bytes && evict_walk.evicted);
+
+ /* One more attempt if we hit the low limit during sleeping-lock pass. */
+ if (lret < target_bytes && evict_walk.hit_low && !evict_walk.try_low) {
+ evict_walk.try_low = true;
+ goto retry;
+ }
+
+out:
+ return lret;
+}
+EXPORT_SYMBOL(ttm_bo_evict_cgroup);
+
/**
* ttm_bo_pin - Pin the buffer object.
* @bo: The buffer object to pin
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index f83b7d5ec6c6..81c6a674c462 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -999,7 +999,8 @@ __ttm_bo_lru_cursor_next(struct ttm_bo_lru_cursor *curs)
bo = res->bo;
if (ttm_lru_walk_trylock(curs, bo))
bo_locked = true;
- else if (!arg->ticket || arg->ctx->no_wait_gpu || arg->trylock_only)
+ else if ((!arg->ticket && !arg->sleeping_lock) || arg->ctx->no_wait_gpu ||
+ arg->trylock_only)
continue;
if (!ttm_bo_get_unless_zero(bo)) {
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 0e5f1582f13d..6d24f3320892 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -950,3 +950,40 @@ void ttm_resource_manager_create_debugfs(struct ttm_resource_manager *man,
#endif
}
EXPORT_SYMBOL(ttm_resource_manager_create_debugfs);
+
+static int ttm_resource_manager_dmem_reclaim(struct dmem_cgroup_pool_state *pool,
+ u64 target_bytes, void *priv)
+{
+ struct ttm_resource_manager *man = priv;
+ struct ttm_operation_ctx ctx = { .interruptible = true };
+ s64 freed;
+
+ freed = ttm_bo_evict_cgroup(man->bdev, man, pool, target_bytes, &ctx);
+ if (freed < 0)
+ return freed;
+
+ return freed >= (s64)target_bytes ? 0 : -ENOSPC;
+}
+
+/**
+ * ttm_resource_manager_set_dmem_region() - Associate a dmem cgroup region with a
+ * resource manager and register a reclaim
+ * callback.
+ * @man: The resource manager.
+ * @region: The dmem cgroup region to associate, may be NULL or IS_ERR().
+ *
+ * Sets @man->cg and registers ttm_resource_manager_dmem_reclaim() so that
+ * writing to dmem.max below current usage triggers TTM eviction rather than
+ * returning -EBUSY to userspace.
+ */
+void ttm_resource_manager_set_dmem_region(struct ttm_resource_manager *man,
+ struct dmem_cgroup_region *region)
+{
+ if (!IS_ERR_OR_NULL(region)) {
+ man->cg = region;
+ dmem_cgroup_region_set_reclaim(region,
+ ttm_resource_manager_dmem_reclaim,
+ man);
+ }
+}
+EXPORT_SYMBOL(ttm_resource_manager_set_dmem_region);
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index 8310bc3d55f9..32791c4db2a9 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -226,6 +226,11 @@ struct ttm_lru_walk_arg {
struct ww_acquire_ctx *ticket;
/** @trylock_only: Only use trylock for locking. */
bool trylock_only;
+ /**
+ * @sleeping_lock: Use sleeping locks even with %NULL @ticket.
+ * @trylock_only has precedence over this field.
+ */
+ bool sleeping_lock;
};
/**
@@ -431,6 +436,11 @@ void ttm_bo_unpin(struct ttm_buffer_object *bo);
int ttm_bo_evict_first(struct ttm_device *bdev,
struct ttm_resource_manager *man,
struct ttm_operation_ctx *ctx);
+s64 ttm_bo_evict_cgroup(struct ttm_device *bdev,
+ struct ttm_resource_manager *man,
+ struct dmem_cgroup_pool_state *limit_pool,
+ s64 target_bytes,
+ struct ttm_operation_ctx *ctx);
int ttm_bo_access(struct ttm_buffer_object *bo, unsigned long offset,
void *buf, int len, int write);
vm_fault_t ttm_bo_vm_reserve(struct ttm_buffer_object *bo,
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index a5d386583fb6..1ba3ae39763c 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -39,6 +39,7 @@
struct dentry;
struct dmem_cgroup_device;
+struct dmem_cgroup_region;
struct drm_printer;
struct ttm_device;
struct ttm_resource_manager;
@@ -477,6 +478,9 @@ void ttm_resource_manager_init(struct ttm_resource_manager *man,
struct ttm_device *bdev,
uint64_t size);
+void ttm_resource_manager_set_dmem_region(struct ttm_resource_manager *man,
+ struct dmem_cgroup_region *region);
+
int ttm_resource_manager_evict_all(struct ttm_device *bdev,
struct ttm_resource_manager *man);
--
2.54.0
^ permalink raw reply related
* [PATCH v4 4/5] drm/xe: Wire up dmem cgroup reclaim for VRAM manager
From: Thomas Hellström @ 2026-05-12 8:24 UTC (permalink / raw)
To: intel-xe
Cc: Thomas Hellström, Natalie Vock, Johannes Weiner, Tejun Heo,
Michal Koutný, cgroups, Huang Rui, Matthew Brost,
Matthew Auld, Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Simona Vetter, David Airlie, Christian König, Alex Deucher,
Rodrigo Vivi, dri-devel, amd-gfx, linux-kernel
In-Reply-To: <20260512082406.44470-1-thomas.hellstrom@linux.intel.com>
Register the VRAM manager with the dmem cgroup reclaim infrastructure
so that lowering dmem.max below current VRAM usage triggers TTM
eviction rather than failing with -EBUSY.
v4:
- Rebased on drm-tip; dropped the XE_PL_STOLEN guard as stolen memory
uses a separate TTM manager and never calls __xe_ttm_vram_mgr_init().
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index b518f7dec680..d8d596d8575b 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -299,14 +299,10 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr,
u64 default_page_size)
{
struct ttm_resource_manager *man = &mgr->manager;
+ struct dmem_cgroup_region *cg;
const char *name;
int err;
- name = mem_type == XE_PL_VRAM0 ? "vram0" : "vram1";
- man->cg = drmm_cgroup_register_region(&xe->drm, name, size);
- if (IS_ERR(man->cg))
- return PTR_ERR(man->cg);
-
man->func = &xe_ttm_vram_mgr_func;
mgr->mem_type = mem_type;
err = drmm_mutex_init(&xe->drm, &mgr->lock);
@@ -317,6 +313,14 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr,
mgr->visible_avail = io_size;
ttm_resource_manager_init(man, &xe->ttm, size);
+
+ name = mem_type == XE_PL_VRAM0 ? "vram0" : "vram1";
+ cg = drmm_cgroup_register_region(&xe->drm, name, size);
+ if (IS_ERR(cg))
+ return PTR_ERR(cg);
+
+ ttm_resource_manager_set_dmem_region(man, cg);
+
err = gpu_buddy_init(&mgr->mm, man->size, default_page_size);
if (err)
return err;
--
2.54.0
^ permalink raw reply related
* [PATCH v4 5/5] drm/amdgpu: Wire up dmem cgroup reclaim for VRAM manager
From: Thomas Hellström @ 2026-05-12 8:24 UTC (permalink / raw)
To: intel-xe
Cc: Thomas Hellström, Natalie Vock, Johannes Weiner, Tejun Heo,
Michal Koutný, cgroups, Huang Rui, Matthew Brost,
Matthew Auld, Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Simona Vetter, David Airlie, Christian König, Alex Deucher,
Rodrigo Vivi, dri-devel, amd-gfx, linux-kernel
In-Reply-To: <20260512082406.44470-1-thomas.hellstrom@linux.intel.com>
Register the VRAM manager with the dmem cgroup reclaim infrastructure
so that lowering dmem.max below current VRAM usage triggers TTM
eviction rather than failing with -EBUSY.
Guard place->flags in amdgpu_ttm_bo_eviction_valuable() against NULL,
as the TTM reclaim path passes a NULL place in cgroup drain mode.
v3:
- Rebased on fix for uninitialized list and buddy allocator on the
drmm_cgroup_register_region() error path.
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 9 ++++++---
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 2740de94e93c..8cbcd33f51a5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1488,7 +1488,7 @@ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
dma_resv_for_each_fence(&resv_cursor, bo->base.resv,
DMA_RESV_USAGE_BOOKKEEP, f) {
if (amdkfd_fence_check_mm(f, current->mm) &&
- !(place->flags & TTM_PL_FLAG_CONTIGUOUS))
+ !(place && (place->flags & TTM_PL_FLAG_CONTIGUOUS)))
return false;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index ac3f71d77140..a1f1ae264a40 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
@@ -916,6 +916,7 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev)
{
struct amdgpu_vram_mgr *mgr = &adev->mman.vram_mgr;
struct ttm_resource_manager *man = &mgr->manager;
+ struct dmem_cgroup_region *cg;
int err;
ttm_resource_manager_init(man, &adev->mman.bdev,
@@ -932,9 +933,11 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev)
if (err)
return err;
- man->cg = drmm_cgroup_register_region(adev_to_drm(adev), "vram", adev->gmc.real_vram_size);
- if (IS_ERR(man->cg))
- return PTR_ERR(man->cg);
+ cg = drmm_cgroup_register_region(adev_to_drm(adev), "vram",
+ adev->gmc.real_vram_size);
+ if (IS_ERR(cg))
+ return PTR_ERR(cg);
+ ttm_resource_manager_set_dmem_region(man, cg);
ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_VRAM, &mgr->manager);
ttm_resource_manager_set_used(man, true);
--
2.54.0
^ permalink raw reply related
* Re: [PATCH v2 00/10] sched: Flatten the pick
From: Vincent Guittot @ 2026-05-12 8:42 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
On Mon, 11 May 2026 at 14:07, Peter Zijlstra <peterz@infradead.org> wrote:
>
> Hi!
>
> So cgroup scheduling has always been a pain in the arse. The problems start
> with weight distribution and end with hierachical picks and it all sucks.
>
> The problems with weight distribution are related to that infernal global
> fraction:
>
> tg->w * grq_i->w
> ge_i->w = ----------------
> \Sum_j grq_j->w
>
> which we've approximated reasonably well by now. However, the immediate
> consequence of this fraction is that the total group weight (tg->w) gets
> fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup
> weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine
> with the fact that 256 CPU systems are relatively common these days, this
> becomes painful.
>
> The common 'solution' is to inflate the group weight by 'nr_cpus'; the
> immediate problem with that is that when all load of a group gets concentrated
> on a single CPU, the per-cpu cgroup weight becomes insanely large, easily
> exceeding nice -20.
>
> Additionally there are numerical limits on the max weight you can have before
> the math starts suffering overflows. As such there is a definite limit on the
> total group weight. Which has annoyed people ;-)
>
> The first few patches add a knob /debug/sched/cgroup_mode and a few different
> options on how to deal with this. My favourite is 'concur', but obviously that
> is also the most expensive one :-/ It adds a tg->tasks counter which makes the
> update_tg_load_avg() thing more expensive.
>
> I have some ideas but I figured I ought to share these things before sinking
> more time into it.
>
>
> On to the hierarchical pick; this has been causing trouble for a very long
> time. So once again an attempt at flatting it. The basic idea is to keep the
> full hierarchical load tracking as-is, but keep all the runnable entities in a
> single level. The immediate concequence of all this is ofcourse that we need to
> constantly re-compute the effective weight of each entity as things progress.
>
> Reweight is done on:
> - enqueue
> - pick -- or rather set_next_entity(.first=true)
> - tick
>
> So while the {en,de}queue operations are still O(depth) due to the full
> accounting mess, the pick is now a single level. Removing the intermediate
> levels that obscure runnability etc.
>
>
> For testing, I've done a little experiment, I dug out what is colloqually known
> as a potato. A trusty old Sandybridge 12600k with a RX 580, and ran a game on
> it. From GOG, I had available 'Shadows: Awakens', a fun title that normally
> runs really well on this machine (provided you stick to 1080p).
>
> To make it interesting, I added 8 (one for each logical CPU) copies of: 'nice
> spin.sh'; this results in the game becoming almost unplayable, as in proper
> terrible.
>
> I used MangoHUD to record a few minutes of playtime for statistics, and then
> quit the came and re-started it with a shorter slice set (base/10). This
> results in the game being entirely playable -- not great, but definiltey
> playable.
>
> Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper)
> Intel Core i7-2600K
> AMD Radeon RX 580
>
> Shadows Awakening (GOG)
>
> default slice(*)
>
> FPS min 3.8 20.6
> avg 48.0 57.2
> mag 87.4 80.3
>
> FT min 9.4 8.4
> avg 34.5 19.5
> max 107.4 37.2
>
> FPS (Frames Per Second)
> FT (FrameTime)
>
> [*] Command prefix: 'chrt -o --sched-runtime 280000 0'
> effectively setting 'base_slice_ns/10'
>
> I have not compared to a kernel without flat on, just wanted to run non trivial
> workloads and play with slice to make sure everything 'works'.
I haven't reviewed the patches yet but I ran some tests with it while
testing sched latency related changes for short slice wakeup
preemption. I have some large hackbench regressions with this series
on HMP system with and without EAS. those figures are unexpected
because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset
slice 2.8ms 16ms 2.8ms 16ms
dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%
radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%
Increasing the slice partly removes regressions but tis is surprising
because the bench runs at root cfs and I thought that results will not
change in such a case
I will review the patchset and try to get what is going wrong
>
>
> Can also be had:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat
>
> include/linux/cpuset.h | 6
> include/linux/sched.h | 1
> kernel/cgroup/cpuset.c | 15
> kernel/sched/core.c | 47 --
> kernel/sched/debug.c | 171 +++++---
> kernel/sched/fair.c | 1038 ++++++++++++++++++++++---------------------------
> kernel/sched/pelt.c | 6
> kernel/sched/sched.h | 44 --
> 8 files changed, 672 insertions(+), 656 deletions(-)
>
> ---
> Change since v1 ( https://patch.msgid.link/20260317095113.387450089@infradead.org ):
> - various Sashiko thingies
> - rebase atop curren -tip
>
>
^ permalink raw reply
* [PATCH] cgroup/cpuset: Return only actually allocated CPUs during partition invalidation
From: Sun Shaojie @ 2026-05-12 9:00 UTC (permalink / raw)
To: Waiman Long, Chen Ridong, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, sunshaojie
From: sunshaojie <sunshaojie@kylinos.cn>
In update_parent_effective_cpumask() with partcmd_invalidate, the CPUs
to return to the parent are computed as:
adding = cpumask_and(tmp->addmask, xcpus, parent->effective_xcpus);
where xcpus = user_xcpus(cs) which returns cs->exclusive_cpus (if set)
or cs->cpus_allowed. When exclusive_cpus is not set, user_xcpus(cs) can
contain CPUs that were never actually granted to the partition due to
sibling exclusion in compute_excpus(). Consequently, the invalidation
may return CPUs to the parent that remain in use by sibling partitions,
causing overlapping effective_cpus and triggering the
WARN_ON_ONCE(1) in generate_sched_domains().
Use cs->effective_xcpus instead, which reflects the CPUs actually
granted to this partition.
Reproducer (on a 4-CPU machine):
cd /sys/fs/cgroup
mkdir a1 b1
# a1 becomes partition root with CPUs 0-1
echo "0-1" > a1/cpuset.cpus
echo "root" > a1/cpuset.cpus.partition
# b1 becomes partition root with CPUs 1-2, but sibling exclusion
# reduces its effective_xcpus to CPU 2 only
echo "1-2" > b1/cpuset.cpus
echo "root" > b1/cpuset.cpus.partition
# b1 changes cpus_allowed to 0-1 -> partition invalidation
echo "0-1" > b1/cpuset.cpus
# Expected: CPUs 2-3 (only CPU 2 returned from b1)
# Actual: CPUs 1-3 (CPU 0-1 returned, overlapping with a1)
cat cpuset.cpus.effective
dmesg will also show a WARNING from generate_sched_domains() reporting
overlapping partition root effective_cpus.
Fixes: 0c7f293efc87 ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2")
Signed-off-by: sunshaojie <sunshaojie@kylinos.cn>
---
kernel/cgroup/cpuset.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 1335e437098e..2311470ef077 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1715,7 +1715,8 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
*/
if (is_partition_valid(parent))
adding = cpumask_and(tmp->addmask,
- xcpus, parent->effective_xcpus);
+ cs->effective_xcpus,
+ parent->effective_xcpus);
if (old_prs > 0)
new_prs = -old_prs;
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v3 03/12] mm/huge_memory: move THP gfp limit helper into header
From: Baolin Wang @ 2026-05-12 9:02 UTC (permalink / raw)
To: Zi Yan, Kairui Song
Cc: linux-mm, Andrew Morton, David Hildenbrand, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <125AABD0-02D5-4656-9F55-4B5BFBD5BD3D@nvidia.com>
On 4/22/26 1:23 AM, Zi Yan wrote:
> On 21 Apr 2026, at 13:21, Kairui Song wrote:
>
>> On Tue, Apr 21, 2026 at 9:14 PM Zi Yan <ziy@nvidia.com> wrote:
>>>
>>> On 21 Apr 2026, at 2:16, Kairui Song via B4 Relay wrote:
>>>
>>>> From: Kairui Song <kasong@tencent.com>
>>>>
>>>> Shmem has some special requirements for THP GFP and has to limit it in
>>>> certain zones or provide a more lenient fallback.
>>>>
>>>> We'll use this helper for generic swap THP allocation, which needs to
>>>> support shmem. For a typical GFP_HIGHUSER_MOVABLE swap-in, this helper
>>>> is basically a no-op. But it's necessary for certain shmem users, mostly
>>>> drivers.
>>>>
>>>> No feature change.
>>>>
>>>> Signed-off-by: Kairui Song <kasong@tencent.com>
>>>> ---
>>>> include/linux/huge_mm.h | 30 ++++++++++++++++++++++++++++++
>>>> mm/shmem.c | 30 +++---------------------------
>>>> 2 files changed, 33 insertions(+), 27 deletions(-)
>>>>
>>>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
>>>> index 2949e5acff35..ffe5a120eee4 100644
>>>> --- a/include/linux/huge_mm.h
>>>> +++ b/include/linux/huge_mm.h
>>>> @@ -237,6 +237,31 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma,
>>>> return true;
>>>> }
>>>>
>>>> +/*
>>>> + * Make sure huge_gfp is always more limited than limit_gfp.
>>>> + * Some shmem users want THP allocation to be done less aggressively
>>>> + * and only in certain zone.
>>>> + */
>>>> +static inline gfp_t thp_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
>>>
>>> Would it be better to rename it to thp_swap_limit_gfp_mask() or something
>>> more descriptive? I am just worried about misuses in the future due to
>>> the generic thp prefix.
>>
>> Good idea, I wasn't sure if this might be helpful for any other user,
>> but for now naming it more descriptive does help to avoid misuse.
>>
>> How about thp_shmem_limit_gfp_mask? Ordinary swap is fine with thp
>> gfp, only shmem is a bit special.
>>
>
> Sounds good to me. Thanks.
Sounds good to me too. Feel free to add:
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
^ permalink raw reply
* [PATCH RFC 0/5] memcg: dma-buf per-cgroup accounting via pid_fd
From: Albert Esteve @ 2026-05-12 9:10 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan
Cc: cgroups, linux-doc, linux-kernel, linux-media, dri-devel,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, Albert Esteve, mripard, echanude
This RFC builds on T.J. Mercier's earlier series [1] which added
a memory.stat counter for exported dma-bufs and a binder-backed
mechanism to transfer charges between cgroups.
The first commit is taken almost verbatim from TJ's series:
it introduces MEMCG_DMABUF as a dedicated per-cgroup stat, so that
the total exported dma-buf footprint is visible both system-wide
(via the root cgroup) and per-application (via per-process cgroups).
This avoids the overhead of DMABUF_SYSFS_STATS and integrates
naturally into the existing cgroup memory hierarchy.
The rest of the series departs from TJ's approach. While the first
commit introduces the memcg stat infrastructure for dmabufs, the
export-time charging it introduces in dma_buf_export() is then
superseded: we charge at dma_heap_ioctl_allocate() time, using a
new charge_pid_fd field in struct dma_heap_allocation_data. The
allocator opens a pidfd for its client (e.g., from binder's
sender_pid), passes it to the ioctl, and the kernel charges the
buffer directly to the client's cgroup at allocation time, so no
transfer step is needed.
This decouples the accounting path from binder entirely:
any allocator that knows its client's PID can use the pid_fd
mechanism regardless of the IPC transport in use.
The cross-cgroup charging capability requires access control.
Patches #3 and #4 add a generic LSM hook (security_dma_heap_alloc)
and an SELinux implementation based on a new dma_heap object class
with a charge_to permission, so policy authors can express which
domains are allowed to charge memory to another domain's cgroup.
Last patch adds some tests to verify the new charge_pid_fd field.
We are sending it as an RFC to spark broader discussion. It may or
may not be the right path forward, and we welcome feedback on the
trade-offs.
Collision note: Eric Chanudet's series [2] adds __GFP_ACCOUNT to
system_heap page allocations as an opt-in module parameter. That
approach charges pages to the allocator's own kmem, which overlaps with
MEMCG_DMABUF. This series explicitly removes __GFP_ACCOUNT from system
heap allocations and routes all accounting through the MEMCG_DMABUF
path to avoid double-counting.
[1] https://lore.kernel.org/cgroups/20230109213809.418135-1-tjmercier@google.com/
[2] https://lore.kernel.org/r/20260113-dmabuf-heap-system-memcg-v2-0-e85722cc2f24@redhat.com
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
Albert Esteve (4):
dma-heap: charge dma-buf memory via explicit memcg
security: dma-heap: Add dma_heap_alloc LSM hook
selinux: Restrict cross-cgroup dma-heap charging
selftests/dmabuf-heaps: Add dma-buf memcg accounting tests
T.J. Mercier (1):
memcg: Track exported dma-buffers
Documentation/admin-guide/cgroup-v2.rst | 5 +
drivers/dma-buf/dma-buf.c | 7 +
drivers/dma-buf/dma-heap.c | 54 +++++-
drivers/dma-buf/heaps/system_heap.c | 2 -
include/linux/dma-buf.h | 4 +
include/linux/lsm_hook_defs.h | 1 +
include/linux/memcontrol.h | 37 ++++
include/linux/security.h | 7 +
include/uapi/linux/dma-heap.h | 6 +
mm/memcontrol.c | 19 ++
security/security.c | 16 ++
security/selinux/hooks.c | 7 +
security/selinux/include/classmap.h | 1 +
tools/testing/selftests/cgroup/Makefile | 2 +-
tools/testing/selftests/cgroup/test_memcontrol.c | 143 +++++++++++++-
tools/testing/selftests/dmabuf-heaps/config | 1 +
tools/testing/selftests/dmabuf-heaps/dmabuf-heap.c | 126 ++++++++++++-
tools/testing/selftests/dmabuf-heaps/vmtest.sh | 205 +++++++++++++++++++++
18 files changed, 633 insertions(+), 10 deletions(-)
---
base-commit: 74fe02ce122a6103f207d29fafc8b3a53de6abaf
change-id: 20260508-v2_20230123_tjmercier_google_com-f44fcfb16530
Best regards,
--
Albert Esteve <aesteve@redhat.com>
^ permalink raw reply
* [PATCH RFC 1/5] memcg: Track exported dma-buffers
From: Albert Esteve @ 2026-05-12 9:10 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan
Cc: cgroups, linux-doc, linux-kernel, linux-media, dri-devel,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, Albert Esteve, mripard, echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-0-6326701c3691@redhat.com>
From: "T.J. Mercier" <tjmercier@google.com>
When a buffer is exported to userspace, use memcg to attribute the
buffer to the allocating cgroup until all buffer references are
released.
Unlike the dmabuf sysfs stats implementation, this memcg accounting
avoids contention over the kernfs_rwsem incurred when creating or
removing nodes.
Signed-off-by: T.J. Mercier <tjmercier@google.com>
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 ++++
drivers/dma-buf/dma-buf.c | 13 ++++++++++++
include/linux/dma-buf.h | 4 ++++
include/linux/memcontrol.h | 37 +++++++++++++++++++++++++++++++++
mm/memcontrol.c | 19 +++++++++++++++++
5 files changed, 77 insertions(+)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6efd0095ed995..8bdbc2e866430 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1635,6 +1635,10 @@ The following nested keys are defined.
Amount of memory used for storing in-kernel data
structures.
+ dmabuf (npn)
+ Amount of memory used for exported DMA buffers allocated by the cgroup.
+ Stays with the allocating cgroup regardless of how the buffer is shared.
+
workingset_refault_anon
Number of refaults of previously evicted anonymous pages.
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index 71f37544a5c61..ce02377f48908 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -14,6 +14,7 @@
#include <linux/fs.h>
#include <linux/slab.h>
#include <linux/dma-buf.h>
+#include <linux/memcontrol.h>
#include <linux/dma-fence.h>
#include <linux/dma-fence-unwrap.h>
#include <linux/anon_inodes.h>
@@ -180,6 +181,9 @@ static void dma_buf_release(struct dentry *dentry)
*/
BUG_ON(dmabuf->cb_in.active || dmabuf->cb_out.active);
+ mem_cgroup_uncharge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
+ mem_cgroup_put(dmabuf->memcg);
+
dmabuf->ops->release(dmabuf);
if (dmabuf->resv == (struct dma_resv *)&dmabuf[1])
@@ -760,6 +764,13 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
dmabuf->resv = resv;
}
+ dmabuf->memcg = get_mem_cgroup_from_mm(current->mm);
+ if (!mem_cgroup_charge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE,
+ GFP_KERNEL)) {
+ ret = -ENOMEM;
+ goto err_memcg;
+ }
+
file->private_data = dmabuf;
file->f_path.dentry->d_fsdata = dmabuf;
dmabuf->file = file;
@@ -770,6 +781,8 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
return dmabuf;
+err_memcg:
+ mem_cgroup_put(dmabuf->memcg);
err_file:
fput(file);
err_module:
diff --git a/include/linux/dma-buf.h b/include/linux/dma-buf.h
index d1203da56fc5f..d9f1ccb51c60e 100644
--- a/include/linux/dma-buf.h
+++ b/include/linux/dma-buf.h
@@ -27,6 +27,7 @@
struct device;
struct dma_buf;
struct dma_buf_attachment;
+struct mem_cgroup;
/**
* struct dma_buf_ops - operations possible on struct dma_buf
@@ -429,6 +430,9 @@ struct dma_buf {
__poll_t active;
} cb_in, cb_out;
+
+ /** @memcg: the cgroup to which this buffer is currently attributed */
+ struct mem_cgroup *memcg;
};
/**
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index dc3fa687759b4..10068a833ad9e 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -39,6 +39,7 @@ enum memcg_stat_item {
MEMCG_ZSWAP_B,
MEMCG_ZSWAPPED,
MEMCG_ZSWAP_INCOMP,
+ MEMCG_DMABUF,
MEMCG_NR_STAT,
};
@@ -649,6 +650,24 @@ int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
gfp_t gfp, swp_entry_t entry);
+/**
+ * mem_cgroup_charge_dmabuf - Charge dma-buf memory to a cgroup and update stat counter
+ * @memcg: memcg to charge
+ * @nr_pages: number of pages to charge
+ * @gfp_mask: reclaim mode
+ *
+ * Charges @nr_pages to @memcg. Returns %true if the charge fit within
+ * @memcg's configured limit, %false if it doesn't.
+ */
+bool __mem_cgroup_charge_dmabuf(struct mem_cgroup *memcg, unsigned int nr_pages, gfp_t gfp_mask);
+static inline bool mem_cgroup_charge_dmabuf(struct mem_cgroup *memcg, unsigned int nr_pages,
+ gfp_t gfp_mask)
+{
+ if (mem_cgroup_disabled())
+ return true;
+ return __mem_cgroup_charge_dmabuf(memcg, nr_pages, gfp_mask);
+}
+
void __mem_cgroup_uncharge(struct folio *folio);
/**
@@ -664,6 +683,14 @@ static inline void mem_cgroup_uncharge(struct folio *folio)
__mem_cgroup_uncharge(folio);
}
+void __mem_cgroup_uncharge_dmabuf(struct mem_cgroup *memcg, unsigned int nr_pages);
+static inline void mem_cgroup_uncharge_dmabuf(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_uncharge_dmabuf(memcg, nr_pages);
+}
+
void __mem_cgroup_uncharge_folios(struct folio_batch *folios);
static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios)
{
@@ -1142,10 +1169,20 @@ static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
return 0;
}
+static inline bool mem_cgroup_charge_dmabuf(struct mem_cgroup *memcg, unsigned int nr_pages,
+ gfp_t gfp_mask)
+{
+ return true;
+}
+
static inline void mem_cgroup_uncharge(struct folio *folio)
{
}
+static inline void mem_cgroup_uncharge_dmabuf(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+}
+
static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios)
{
}
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c03d4787d4668..15cee13d3ccd6 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -433,6 +433,7 @@ static const unsigned int memcg_stat_items[] = {
MEMCG_ZSWAP_B,
MEMCG_ZSWAPPED,
MEMCG_ZSWAP_INCOMP,
+ MEMCG_DMABUF,
};
#define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items)
@@ -1580,6 +1581,7 @@ static const struct memory_stat memory_stats[] = {
#ifdef CONFIG_HUGETLB_PAGE
{ "hugetlb", NR_HUGETLB },
#endif
+ { "dmabuf", MEMCG_DMABUF },
/* The memory events */
{ "workingset_refault_anon", WORKINGSET_REFAULT_ANON },
@@ -5399,6 +5401,23 @@ void mem_cgroup_flush_workqueue(void)
flush_workqueue(memcg_wq);
}
+bool __mem_cgroup_charge_dmabuf(struct mem_cgroup *memcg, unsigned int nr_pages, gfp_t gfp_mask)
+{
+ if (try_charge(memcg, gfp_mask, nr_pages) == 0) {
+ mod_memcg_state(memcg, MEMCG_DMABUF, nr_pages);
+ return true;
+ }
+
+ return false;
+}
+
+void __mem_cgroup_uncharge_dmabuf(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ mod_memcg_state(memcg, MEMCG_DMABUF, -nr_pages);
+ if (!mem_cgroup_is_root(memcg))
+ refill_stock(memcg, nr_pages);
+}
+
static int __init cgroup_memory(char *s)
{
char *token;
--
2.53.0
^ permalink raw reply related
* [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Albert Esteve @ 2026-05-12 9:10 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan
Cc: cgroups, linux-doc, linux-kernel, linux-media, dri-devel,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, Albert Esteve, mripard, echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-0-6326701c3691@redhat.com>
On embedded platforms a central process often allocates dma-buf
memory on behalf of client applications. Without a way to
attribute the charge to the requesting client's cgroup, the
cost lands on the allocator, making per-cgroup memory limits
ineffective for the actual consumers.
Add charge_pid_fd to struct dma_heap_allocation_data. When set to
a valid pidfd, DMA_HEAP_IOCTL_ALLOC resolves the target task's
memcg and charges the buffer there via mem_cgroup_charge_dmabuf()
inside dma_heap_buffer_alloc(). Without charge_pid_fd, and with
the mem_accounting module parameter enabled, the buffer is charged
to the allocator's own cgroup.
Additionally, commit 3c227be90659 ("dma-buf: system_heap: account for
system heap allocation in memcg") adds __GFP_ACCOUNT to system-heap
page allocations. Keeping __GFP_ACCOUNT would charge the same pages
twice (once to kmem, once to MEMCG_DMABUF), thus remove it and route
all accounting through a single MEMCG_DMABUF path.
Usage examples:
1. Central allocator charging to a client at allocation time.
The allocator knows the client's PID (e.g., from binder's
sender_pid) and uses pidfd to attribute the charge:
pid_t client_pid = txn->sender_pid;
int pidfd = pidfd_open(client_pid, 0);
struct dma_heap_allocation_data alloc = {
.len = buffer_size,
.fd_flags = O_RDWR | O_CLOEXEC,
.charge_pid_fd = pidfd,
};
ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
close(pidfd);
/* alloc.fd is now charged to client's cgroup */
2. Default allocation (no pidfd, mem_accounting=1).
When charge_pid_fd is not set and the mem_accounting module
parameter is enabled, the buffer is charged to the allocator's
own cgroup:
struct dma_heap_allocation_data alloc = {
.len = buffer_size,
.fd_flags = O_RDWR | O_CLOEXEC,
};
ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
/* charged to current process's cgroup */
Current limitations:
- Single-owner model: a dma-buf carries one memcg charge regardless of
how many processes share it. Means only the first owner (and exporter)
of the shared buffer bears the charge.
- Only memcg accounting supported. While this makes sense for system
heap buffers, other heaps (e.g., CMA heaps) will require selectively
charging also for the dmem controller.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
Documentation/admin-guide/cgroup-v2.rst | 5 ++--
drivers/dma-buf/dma-buf.c | 16 ++++---------
drivers/dma-buf/dma-heap.c | 42 ++++++++++++++++++++++++++++++---
drivers/dma-buf/heaps/system_heap.c | 2 --
include/uapi/linux/dma-heap.h | 6 +++++
5 files changed, 53 insertions(+), 18 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 8bdbc2e866430..824d269531eb1 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1636,8 +1636,9 @@ The following nested keys are defined.
structures.
dmabuf (npn)
- Amount of memory used for exported DMA buffers allocated by the cgroup.
- Stays with the allocating cgroup regardless of how the buffer is shared.
+ Amount of memory used for exported DMA buffers allocated by or on
+ behalf of the cgroup. Stays with the allocating cgroup regardless
+ of how the buffer is shared.
workingset_refault_anon
Number of refaults of previously evicted anonymous pages.
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index ce02377f48908..23fb758b78297 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -181,8 +181,11 @@ static void dma_buf_release(struct dentry *dentry)
*/
BUG_ON(dmabuf->cb_in.active || dmabuf->cb_out.active);
- mem_cgroup_uncharge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
- mem_cgroup_put(dmabuf->memcg);
+ if (dmabuf->memcg) {
+ mem_cgroup_uncharge_dmabuf(dmabuf->memcg,
+ PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
+ mem_cgroup_put(dmabuf->memcg);
+ }
dmabuf->ops->release(dmabuf);
@@ -764,13 +767,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
dmabuf->resv = resv;
}
- dmabuf->memcg = get_mem_cgroup_from_mm(current->mm);
- if (!mem_cgroup_charge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE,
- GFP_KERNEL)) {
- ret = -ENOMEM;
- goto err_memcg;
- }
-
file->private_data = dmabuf;
file->f_path.dentry->d_fsdata = dmabuf;
dmabuf->file = file;
@@ -781,8 +777,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
return dmabuf;
-err_memcg:
- mem_cgroup_put(dmabuf->memcg);
err_file:
fput(file);
err_module:
diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
index ac5f8685a6494..ff6e259afcdc0 100644
--- a/drivers/dma-buf/dma-heap.c
+++ b/drivers/dma-buf/dma-heap.c
@@ -7,13 +7,17 @@
*/
#include <linux/cdev.h>
+#include <linux/cgroup.h>
#include <linux/device.h>
#include <linux/dma-buf.h>
#include <linux/dma-heap.h>
+#include <linux/memcontrol.h>
+#include <linux/sched/mm.h>
#include <linux/err.h>
#include <linux/export.h>
#include <linux/list.h>
#include <linux/nospec.h>
+#include <linux/pidfd.h>
#include <linux/syscalls.h>
#include <linux/uaccess.h>
#include <linux/xarray.h>
@@ -55,10 +59,12 @@ MODULE_PARM_DESC(mem_accounting,
"Enable cgroup-based memory accounting for dma-buf heap allocations (default=false).");
static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
- u32 fd_flags,
- u64 heap_flags)
+ u32 fd_flags, u64 heap_flags,
+ struct mem_cgroup *charge_to)
{
struct dma_buf *dmabuf;
+ unsigned int nr_pages;
+ struct mem_cgroup *memcg = charge_to;
int fd;
/*
@@ -73,6 +79,22 @@ static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
if (IS_ERR(dmabuf))
return PTR_ERR(dmabuf);
+ nr_pages = len / PAGE_SIZE;
+
+ if (memcg)
+ css_get(&memcg->css);
+ else if (mem_accounting)
+ memcg = get_mem_cgroup_from_mm(current->mm);
+
+ if (memcg) {
+ if (!mem_cgroup_charge_dmabuf(memcg, nr_pages, GFP_KERNEL)) {
+ mem_cgroup_put(memcg);
+ dma_buf_put(dmabuf);
+ return -ENOMEM;
+ }
+ dmabuf->memcg = memcg;
+ }
+
fd = dma_buf_fd(dmabuf, fd_flags);
if (fd < 0) {
dma_buf_put(dmabuf);
@@ -102,6 +124,9 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
{
struct dma_heap_allocation_data *heap_allocation = data;
struct dma_heap *heap = file->private_data;
+ struct mem_cgroup *memcg = NULL;
+ struct task_struct *task;
+ unsigned int pidfd_flags;
int fd;
if (heap_allocation->fd)
@@ -113,9 +138,20 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS)
return -EINVAL;
+ if (heap_allocation->charge_pid_fd) {
+ task = pidfd_get_task(heap_allocation->charge_pid_fd, &pidfd_flags);
+ if (IS_ERR(task))
+ return PTR_ERR(task);
+
+ memcg = get_mem_cgroup_from_mm(task->mm);
+ put_task_struct(task);
+ }
+
fd = dma_heap_buffer_alloc(heap, heap_allocation->len,
heap_allocation->fd_flags,
- heap_allocation->heap_flags);
+ heap_allocation->heap_flags,
+ memcg);
+ mem_cgroup_put(memcg);
if (fd < 0)
return fd;
diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
index 03c2b87cb1112..95d7688167b93 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -385,8 +385,6 @@ static struct page *alloc_largest_available(unsigned long size,
if (max_order < orders[i])
continue;
flags = order_flags[i];
- if (mem_accounting)
- flags |= __GFP_ACCOUNT;
page = alloc_pages(flags, orders[i]);
if (!page)
continue;
diff --git a/include/uapi/linux/dma-heap.h b/include/uapi/linux/dma-heap.h
index a4cf716a49fa6..e02b0f8cbc6a1 100644
--- a/include/uapi/linux/dma-heap.h
+++ b/include/uapi/linux/dma-heap.h
@@ -29,6 +29,10 @@
* handle to the allocated dma-buf
* @fd_flags: file descriptor flags used when allocating
* @heap_flags: flags passed to heap
+ * @charge_pid_fd: optional pidfd of the process whose cgroup should be
+ * charged for this allocation; 0 means charge the calling
+ * process's cgroup
+ * @__padding: reserved, must be zero
*
* Provided by userspace as an argument to the ioctl
*/
@@ -37,6 +41,8 @@ struct dma_heap_allocation_data {
__u32 fd;
__u32 fd_flags;
__u64 heap_flags;
+ __u32 charge_pid_fd;
+ __u32 __padding;
};
#define DMA_HEAP_IOC_MAGIC 'H'
--
2.53.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox