Re: [regression] cpuset: offlined CPUs removed from affinity masks

[Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org>]

----- On Feb 19, 2020, at 10:52 AM, Tejun Heo tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org wrote:

> On Wed, Feb 19, 2020 at 10:50:35AM -0500, Mathieu Desnoyers wrote:
>> I can look into figuring out the commit introducing this issue, which I
>> suspect will be close to the introduction of CONFIG_CPUSET into the
>> kernel (which was ages ago). I'll check and let you know.
> 
> Oh, yeah, I'm pretty sure it goes way back. I don't think tracking
> that down would be necessary. I was just wondering whether it was a
> recent change because you said it was a regression.

It's most likely not a recent regression, but it has unfortunate effects
on the affinity mask which directly affects my ongoing work on the
pin_on_cpu() system call [1].

The sched_setaffinity vs cpu hotplug semantic provided by CONFIG_CPUSET=n
if fine for the needs on pin_on_cpu(): when a CPU comes back online,
those reappear in the affinity mask, but it is not the case with
CONFIG_CPUSET=y.

I wonder if applying the online cpu masks to the per-thread affinity mask
is the correct approach ? I suspect what we may be looking for here is to keep
the affinity mask independent of cpu hotplug, and look-up both the per-thread
affinity mask and the online cpu mask whenever the scheduler needs to perform
"is_cpu_allowed()" to check task placement.

Then whenever sched_getaffinity or cpusets try to query the current set of
cpus on which a task can run right now, it could also look at both the task's
affinity mask and the online cpu mask.

Thanks,

Mathieu

[1] https://lore.kernel.org/r/20200121160312.26545-1-mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com