Fwd: Question about how to fix numabalancing to work with isolated cpus?

Fwd: Question about how to fix numabalancing to work with isolated cpus?

[guanjing (D)]

Hi,
     I notice a problem that you guys discussed in the link below:

     https://lore.kernel.org/all/1491464210.4718.82.camel@gmx.de/T/
     "The isolated cpus are part of the cpus allowed list. In the above 
case,
      numabalancing ends up scheduling some of these tasks on isolated 
cpus. "

     I wonder if we're going to fix this eventually?

Thanks,
Guan Jing

Re: Fwd: Question about how to fix numabalancing to work with isolated cpus?

[Mel Gorman]

On Fri, Aug 05, 2022 at 02:42:18PM +0800, guanjing (D) wrote:
> On 2022/6/25 14:33, guanjing (D) wrote:
> > Hi,
> >     I notice a problem that you guys discussed in the link below:
> > 
> >     https://lore.kernel.org/all/1491464210.4718.82.camel@gmx.de/T/
> >     "The isolated cpus are part of the cpus allowed list. In the above
> > case,
> >      numabalancing ends up scheduling some of these tasks on isolated
> > cpus. "
> > 
> >     I wonder if we're going to fix this eventually?
> > 

The last discussion did not reach a solid conclusion. Isolated CPUs
are avoided by the general CPU scheduler but if a task explicitly uses
sched_setaffinity() to include the isolated CPUs then by definition it's
allowed to use those CPUs and NUMA balancing will use allowed CPUs to
improve locality if possible. It's not clear that this is a bug at all
other than it may be surprising in some cases. Furthermore, a task that
is sensitive to latency probably should be using mempolicies to avoid NUMA
balancing entirely as it's a source of interference.

There needs to be a use case that describes exactly what the sematics should
be for a task using isolated CPUs and why. For example, if the task policy
only allowed isolated CPUs then NUMA balancing may never be able to select
a CPU with task_numa_find_cpu, is that expected? Should it be the case
that NUMA balancing avoids isolated CPUs but will select one if no other
local CPU is available? If so, why? If only isolated CPUs are available,
is it expected and desired that data will be migrated instead? If a task is
running on an isolated CPU, should NUMA balancing avoid moving the task off
the CPU? If so, why? Should the behaviour be that if a task is allowed to
run on isolated CPUs, should NUMA balancing simply be disabled? If so, why?
Whatever the result, the sematics should be mentioned under the isocpus=
parameter with an additional note under numa_balancing=.

If anything is to change here, it needs to be made clear why it's a bug
that NUMA balancing selects an allowed but isolated CPU and a description
of what the desired semantics should be.

-- 
Mel Gorman
SUSE Labs