Re: Questions about replacing isolcpus by cgroup-v2

[Frederic Weisbecker <frederic@kernel.org>]

On Fri, Nov 12, 2021 at 04:36:56PM +0100, Sebastian Andrzej Siewior wrote:
> On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:
> > Dear subscribers,
> Hi,
> 
> I Cced cgroups@vger since thus question fits there better.
> I Cced Frederic in case he has come clues regarding isolcpus and
> cgroups.
> 
> > we are currently evaluating how to rework realtime tuning to use cgroup-v2 cpusets instead of the isolcpus kernel parameter.
> > Our use-case are realtime applications with rt and non-rt threads. Hereby, the non-rt thread might create additional non-rt threads:
> > 
> > Example (RT CPU=1, 4 CPUs):
> > - Non-RT Thread (A) with default affinity 0xD (1101b)
> > - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
> > 
> > When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly works:
> > Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a default affinity of 0xD.
> > By that, independent of the threads' priorities, we can ensure that nothing is scheduled on our RT cpu (except from kernel threads, etc...).
> > 
> > During this journey, we discovered the following:
> > 
> > Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
> > When activating the cpuset controller on a cgroup (for the first time), all default CPU affinities are reset.
> > By that, also the default affinity is set to 0xFFFF..., while with isolcpus we expect it to be (0xFFFF - isolcpus).
> > This breaks the example from above, as now the non-RT thread can also be
> > scheduled on the RT CPU.

That sounds buggy from the cpuset-v2 side (adding the maintainers in Cc).

Also please have a look into "[PATCH v8 0/6] cgroup/cpuset: Add new cpuset
partition type & empty effecitve cpus":

	  https://lore.kernel.org/lkml/20211018143619.205065-1-longman@redhat.com/

This stuff adds support for a new "isolated" partition type on cpuset/cgroup-v2
which should behave just like isolcpus.

> > 
> > When only using cgroup-v2, we can isolate our RT process by placing it in a cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
> > However, we do not know of a strategy to set a default affinity:
> > Given the example above, we have no way to ensure that newly created threads are born with an affinity of just 0x2 (without changing the application).
> > 
> > Finally, isolcpus itself is deprecated since kernel 5.4.
> 
> Where is this the deprecation of isolcpus announced/ written?

We tried to deprecate it but too many people are still using it. Better pick an
interface that allows you to change the isolated set at runtime like
cpuset.sched_load_balance on cpuset/cgroup-v1 or the above patchset on v2.

Thanks.