Re: Questions about replacing isolcpus by cgroup-v2

Sebastian Andrzej Siewior
2021-11-12T15:36:56Z

On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:
> Dear subscribers,
Hi,

I Cced cgroups@vger since thus question fits there better.
I Cced Frederic in case he has come clues regarding isolcpus and
cgroups.

> we are currently evaluating how to rework realtime tuning to use cgroup-v2 cpusets instead of the isolcpus kernel parameter.
> Our use-case are realtime applications with rt and non-rt threads. Hereby, the non-rt thread might create additional non-rt threads:
>
> Example (RT CPU=1, 4 CPUs):
> - Non-RT Thread (A) with default affinity 0xD (1101b)
> - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
>
> When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly works:
> Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a default affinity of 0xD.
> By that, independent of the threads' priorities, we can ensure that nothing is scheduled on our RT cpu (except from kernel threads, etc...).
>
> During this journey, we discovered the following:
>
> Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
> When activating the cpuset controller on a cgroup (for the first time), all default CPU affinities are reset.
> By that, also the default affinity is set to 0xFFFF..., while with isolcpus we expect it to be (0xFFFF - isolcpus).
> This breaks the example from above, as now the non-RT thread can also be scheduled on the RT CPU.
>
> When only using cgroup-v2, we can isolate our RT process by placing it in a cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
> However, we do not know of a strategy to set a default affinity:
> Given the example above, we have no way to ensure that newly created threads are born with an affinity of just 0x2 (without changing the application).
>
> Finally, isolcpus itself is deprecated since kernel 5.4.

Where is this the deprecation of isolcpus announced/ written?

> Questions:
>
> 1. What is the best strategy to "isolcpus" similar semantics with cgroups-v2?
> 2. Is there a way to specify the default affinity (within a cgroup)
>
> We are currently at a point where we would write patches to add a default affinity feature to cpusets of cgroupv2.
> But maybe that is not needed or would be the wrong direction, so we wanted to discuss first.
>
> Best regards,
> Felix Mößbauer
> Siemens AG

Sebastian