Re: [PATCH 0/4] sched/numa: Add per-process automatic NUMA balancing control

[Tim Chen <tim.c.chen@linux.intel.com>]

On Fri, 2026-10-16 at 16:18 +0800, Yafang Shao wrote:
> On Fri, Oct 16, 2026 at 4:01 PM Chen, Yu C <yu.c.chen@intel.com> wrote:
> > 
> > On 10/16/2026 3:38 PM, Yafang Shao wrote:
> > > On Fri, Oct 16, 2026 at 1:51 PM Chen Yu <yu.c.chen@intel.com> wrote:
> > > > A more complex scenario is when cgroup A and cgroup B become threaded domains:
> > > > 
> > > >                           root
> > > >                          /    \
> > > >                         X      Y
> > > >                         |      |
> > > >                         A      B          A, B: threaded domains (thread roots)
> > > >                        / \    / \         separate memory.max / io.max
> > > >                      A1   A2 B1  B2       threaded cgroups
> > > >                      ^        ^
> > > >                      |        |
> > > >                      +--------+
> > > >                    want to share one cache group
> > > > 
> > > > In the above case, we might need to change the deployed cgroup hierarchy.
> > > 
> > > I think we can enable LLC-aware scheduling for both A1 and A2 — does
> > > that sound right?
> > > 
> > 
> > Here A1 wants to share data with B1, but A2 does not want to share data
> > with B2 nor B1 nor A1. Enabling cache-aware for A1 and A2 might not help
> > in this case.
> 
> I'm not sure whether this is a real use case or merely a theoretical
> one. I'd suggest we not focus too much on purely theoretical scenarios
> unless someone can demonstrate a practical need.

In our discussions with Vern Hao from Tencent, they have use cases where
A1 and B1 are dealing with data compressions while A2 and B2 are dealing
with databases.  So data are mostly shared between A1 and B1, A2 and B2
due to functionality.  We may put the parent A or B in the same sched group,
but the sched group may contain too many threads to fix in a LLC.

The tricky thing is if there are scenarios where
we want to do grouping one way when we consider resource
usage but another way when we consider data sharing.
 
> 
> > 
> > > > 
> > > > Suppose threads in A1 and threads in A2 belong to the same process Pa,
> > > > this reminds me of the cgroup bpf NUMA balancing case: after enabling
> > > > NUMA balancing for A1 but not setting it for A2, what would be the final
> > > > NUMA balancing decision for process Pa, is it enabled or disabled?
> > > > The NUMA balancing scan/invalid seems to be per process, so if we change
> > > > the setting of its threads, the whole process will be impacted. Or maybe
> > > > we should not enable NUMA balancing for threaded cgroups? Anyway I haven't
> > > > digged into your cgroup bpf patch set yet, let me try to look at it.
> > > 
> > > cgroup NUMAB works the same way: the setting is per cgroup and opt-in.
> > > For Pa in your example, since its threads spread across A1 and A2, you
> > > would enable it on both — enabling it on only one of them is not
> > > meaningful.
> > > 
> > 
> > I see. I haven't yet digged into your code, is there a per cgroup flag
> > cgroup.numab.enabled?
> 
> Right. I will introduce a cgroup::numab_mode field, but without a
> cgroupfs interface file; it will be written through a BPF program.
> 
> > Would there be a logic in the kernel, when
> > A1.numab.enabled
> > is written to true, the kernel detects all the other cgroups {X} that
> > contains threads
> > share the same mm_struct as A1's threads/tasks, and then update {X}'s
> > numab.enabled
> > to the same value as A1, to keep consistency?
> 
> We have no plan to support threaded cgroups for now. If a real need
> arises, the BPF program can be extended to handle it, which would just
> require more kfuncs.
> 
> > In summary, if threads are
> > spread
> > cross different cgroups, when you set the enable flag for 1 cgroup, will the
> > kernel iterates all the other cgroups to update their flags as well? Or
> > leverage userspace to do that?
> 
> I believe this should be implemented in userspace, not automatically
> in the kernel. Use cases are always complex, and flexibility is
> important here.

I think there is a misunderstanding that using prctl necessarily mean
we have to modify the user programs.

With the prctl that Chen Yu and I proposed for cache aware scheduling
[1], the application does not have to change at all. The prctl can be
issued by an orchestrator daemon on other tasks:
the caller passes a target pid and a pid_type (thread,
thread group or process group), and the permission check is       
ptrace_may_access(), as in core scheduling. So a daemon run by the  
platform team can put the application's threads into the right cache                          
group from outside:                            
                                                                                             
  prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_CREATE, leader, 0, PIDTYPE_TGID);     
  prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, tid, leader, PIDTYPE_PID);                 
                                                                                                                                                                                                                       
The daemon can take its policy from wherever is convenient: cgroup 
membership, the pod spec, process names, or a config file. Changing 
the policy means changing the daemon, which the same team owns. That
is the same round-trip as changing a BPF program, and no application
team is involved. This is how schedqos already uses the core                                  
scheduling prctl.                              

That said, we may start with a per cgroup control of
cache aware scheduling that sit next to the prctl [1]
without either one replacing the other.  A lot of the
code in [1] can be reused with per cgroup control
of cache aware scheduling.

It starts small, in the form Tejun suggested for NUMA balancing
in your series [2], and grows only when a real need shows up.

The rule for every step with a BPF based cgroup interface:
the per-task p->sched_cache_grp from [1] 
is the only thing the scheduler reads. Each interface is just
another way to set it, so the scheduler side never changes.

Step 1: a per-cgroup setting
----------------------------

  - struct cgroup gets a sched_cache_grp pointer and a mode:
    inherit (default), off, or "this subtree is one group".
  - One kfunc sets the mode by cgroup id. No cgroupfs file for now.
  - The kernel puts tasks in the cgroup's group:
      - when the mode is set: allocate the group, publish it, then
        walk the cgroup's tasks
      - at fork: in sched_post_fork(), right after cgroup_post_fork()
      - at exec: exec_mmap() installs the cgroup's group instead of the
        mm's one
      - on migration: in cgroup_migrate_execute(), next to the
        ss->attach() calls
  - A task that was grouped explicitly (by the prctl, or later by BPF)
    keeps its group. Cgroup changes do not override it.
  - In the "advise" mode the cgroup's group is created enabled, since
    a new group starts disabled there.

This reuses most of [1]: the refcounted sched_cache_group (patches
1-3), __sched_cache_set() with the pi_lock swap and the footprint
transfer (patch 4), the fork and exec handling (patches 2 and 4), the
per-group enabled flag (patch 5) and the always/advise/never mode
(patch 6). Only the prctl UAPI itself is not needed.

This covers the the case where the sched groups match well with
cgroups: cache aware scheduling can be turned on for group A
only. The other part, a footprint threshold for A that differs from
the global llc_aggr_tolerance, would need a per-group tolerance that
the per-cgroup setting can carry.

Step 2: BPF picks the group
---------------------------

Add kfuncs to create, read, set and reset a group (thin wrappers around
the same code), and one that stores a BPF-chosen group in a cgroup.
The kernel still installs tasks at fork, exec and migration as in step
1. The difference is that two cgroups can now share one group, for
example A1 and B1 in one sched group in the above example.

Step 3: BPF decides per task
----------------------------

Only if someone needs it: programs on the fork, exec and cgroup-attach
hooks that set a task's group directly, for example to split a cgroup
that is too big for one LLC, or to group only some of its tasks. The
set kfunc takes pi_lock, so it needs a context filter that allows it
only in sleepable programs and those hooks.

Each step stands on its own. If step 1 is all anyone ever needs, we
stop there. The prctl stays for users who do not run cgroups or schedqos.

Does step 1 match what you have in mind for the LLC-aware scheduler
on your servers?

[1] https://lore.kernel.org/lkml/cover.1787955777.git.tim.c.chen@linux.intel.com/
[2] https://lore.kernel.org/lkml/20261012120438.52849-1-laoar.shao@gmail.com/

Thanks.

Tim