RE: Questions about replacing isolcpus by cgroup-v2

Moessbauer, Felix
2021-11-12T15:45:12Z

Hi Sebastian,

> -----Original Message-----
> From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Sent: Friday, November 12, 2021 4:37 PM
> To: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@siemens.com>;
> cgroups@vger.kernel.org
> Cc: linux-rt-users@vger.kernel.org; Schild, Henning (T RDA IOT SES-DE)
> <henning.schild@siemens.com>; Kiszka, Jan (T RDA IOT)
> <jan.kiszka@siemens.com>; Schmidt, Adriaan (T RDA IOT SES-DE)
> <adriaan.schmidt@siemens.com>; Frederic Weisbecker <frederic@kernel.org>
> Subject: Re: Questions about replacing isolcpus by cgroup-v2
>
> On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:
> > Dear subscribers,
> Hi,
>
> I Cced cgroups@vger since thus question fits there better.
> I Cced Frederic in case he has come clues regarding isolcpus and cgroups.

Indeed. Thanks!

>
> > we are currently evaluating how to rework realtime tuning to use cgroup-v2
> cpusets instead of the isolcpus kernel parameter.
> > Our use-case are realtime applications with rt and non-rt threads. Hereby, the
> non-rt thread might create additional non-rt threads:
> >
> > Example (RT CPU=1, 4 CPUs):
> > - Non-RT Thread (A) with default affinity 0xD (1101b)
> > - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
> >
> > When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly
> works:
> > Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a
> default affinity of 0xD.
> > By that, independent of the threads' priorities, we can ensure that nothing is
> scheduled on our RT cpu (except from kernel threads, etc...).
> >
> > During this journey, we discovered the following:
> >
> > Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
> > When activating the cpuset controller on a cgroup (for the first time), all
> default CPU affinities are reset.
> > By that, also the default affinity is set to 0xFFFF..., while with isolcpus we
> expect it to be (0xFFFF - isolcpus).
> > This breaks the example from above, as now the non-RT thread can also be
> scheduled on the RT CPU.
> >
> > When only using cgroup-v2, we can isolate our RT process by placing it in a
> cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
> > However, we do not know of a strategy to set a default affinity:
> > Given the example above, we have no way to ensure that newly created
> threads are born with an affinity of just 0x2 (without changing the application).
> >
> > Finally, isolcpus itself is deprecated since kernel 5.4.
>
> Where is this the deprecation of isolcpus announced/ written?

https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
isolcpus= [KNL,SMP,ISOL] Isolate a given set of CPUs from disturbance.
[Deprecated - use cpusets instead]
Format: [flag-list,]<cpu-list>

>
> > Questions:
> >
> > 1. What is the best strategy to "isolcpus" similar semantics with cgroups-v2?
> > 2. Is there a way to specify the default affinity (within a cgroup)
> >
> > We are currently at a point where we would write patches to add a default
> affinity feature to cpusets of cgroupv2.
> > But maybe that is not needed or would be the wrong direction, so we wanted
> to discuss first.
> >
> > Best regards,
> > Felix Mößbauer
> > Siemens AG
>
> Sebastian

Best regards,
Felix