Questions about replacing isolcpus by cgroup-v2

Questions about replacing isolcpus by cgroup-v2

[Moessbauer, Felix]

Dear subscribers,

we are currently evaluating how to rework realtime tuning to use cgroup-v2 cpusets instead of the isolcpus kernel parameter.
Our use-case are realtime applications with rt and non-rt threads. Hereby, the non-rt thread might create additional non-rt threads:

Example (RT CPU=1, 4 CPUs):
- Non-RT Thread (A) with default affinity 0xD (1101b)
- RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)

When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly works:
Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a default affinity of 0xD.
By that, independent of the threads' priorities, we can ensure that nothing is scheduled on our RT cpu (except from kernel threads, etc...).

During this journey, we discovered the following:

Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
When activating the cpuset controller on a cgroup (for the first time), all default CPU affinities are reset.
By that, also the default affinity is set to 0xFFFF..., while with isolcpus we expect it to be (0xFFFF - isolcpus).
This breaks the example from above, as now the non-RT thread can also be scheduled on the RT CPU.

When only using cgroup-v2, we can isolate our RT process by placing it in a cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
However, we do not know of a strategy to set a default affinity:
Given the example above, we have no way to ensure that newly created threads are born with an affinity of just 0x2 (without changing the application).

Finally, isolcpus itself is deprecated since kernel 5.4.

Questions:

1. What is the best strategy to "isolcpus" similar semantics with cgroups-v2?
2. Is there a way to specify the default affinity (within a cgroup)

We are currently at a point where we would write patches to add a default affinity feature to cpusets of cgroupv2.
But maybe that is not needed or would be the wrong direction, so we wanted to discuss first.

Best regards,
Felix Mößbauer
Siemens AG

Re: Questions about replacing isolcpus by cgroup-v2

[Sebastian Andrzej Siewior]

On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:
> Dear subscribers,
Hi,

I Cced cgroups@vger since thus question fits there better.
I Cced Frederic in case he has come clues regarding isolcpus and
cgroups.

> we are currently evaluating how to rework realtime tuning to use cgroup-v2 cpusets instead of the isolcpus kernel parameter.
> Our use-case are realtime applications with rt and non-rt threads. Hereby, the non-rt thread might create additional non-rt threads:
> 
> Example (RT CPU=1, 4 CPUs):
> - Non-RT Thread (A) with default affinity 0xD (1101b)
> - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
> 
> When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly works:
> Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a default affinity of 0xD.
> By that, independent of the threads' priorities, we can ensure that nothing is scheduled on our RT cpu (except from kernel threads, etc...).
> 
> During this journey, we discovered the following:
> 
> Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
> When activating the cpuset controller on a cgroup (for the first time), all default CPU affinities are reset.
> By that, also the default affinity is set to 0xFFFF..., while with isolcpus we expect it to be (0xFFFF - isolcpus).
> This breaks the example from above, as now the non-RT thread can also be scheduled on the RT CPU.
> 
> When only using cgroup-v2, we can isolate our RT process by placing it in a cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
> However, we do not know of a strategy to set a default affinity:
> Given the example above, we have no way to ensure that newly created threads are born with an affinity of just 0x2 (without changing the application).
> 
> Finally, isolcpus itself is deprecated since kernel 5.4.

Where is this the deprecation of isolcpus announced/ written?

> Questions:
> 
> 1. What is the best strategy to "isolcpus" similar semantics with cgroups-v2?
> 2. Is there a way to specify the default affinity (within a cgroup)
> 
> We are currently at a point where we would write patches to add a default affinity feature to cpusets of cgroupv2.
> But maybe that is not needed or would be the wrong direction, so we wanted to discuss first.
> 
> Best regards,
> Felix Mößbauer
> Siemens AG

Sebastian

RE: Questions about replacing isolcpus by cgroup-v2

[Moessbauer, Felix]

Hi Sebastian,

> -----Original Message-----
> From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Sent: Friday, November 12, 2021 4:37 PM
> To: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@siemens.com>;
> cgroups@vger.kernel.org
> Cc: linux-rt-users@vger.kernel.org; Schild, Henning (T RDA IOT SES-DE)
> <henning.schild@siemens.com>; Kiszka, Jan (T RDA IOT)
> <jan.kiszka@siemens.com>; Schmidt, Adriaan (T RDA IOT SES-DE)
> <adriaan.schmidt@siemens.com>; Frederic Weisbecker <frederic@kernel.org>
> Subject: Re: Questions about replacing isolcpus by cgroup-v2
> 
> On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:
> > Dear subscribers,
> Hi,
> 
> I Cced cgroups@vger since thus question fits there better.
> I Cced Frederic in case he has come clues regarding isolcpus and cgroups.

Indeed. Thanks!

> 
> > we are currently evaluating how to rework realtime tuning to use cgroup-v2
> cpusets instead of the isolcpus kernel parameter.
> > Our use-case are realtime applications with rt and non-rt threads. Hereby, the
> non-rt thread might create additional non-rt threads:
> >
> > Example (RT CPU=1, 4 CPUs):
> > - Non-RT Thread (A) with default affinity 0xD (1101b)
> > - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
> >
> > When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly
> works:
> > Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a
> default affinity of 0xD.
> > By that, independent of the threads' priorities, we can ensure that nothing is
> scheduled on our RT cpu (except from kernel threads, etc...).
> >
> > During this journey, we discovered the following:
> >
> > Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
> > When activating the cpuset controller on a cgroup (for the first time), all
> default CPU affinities are reset.
> > By that, also the default affinity is set to 0xFFFF..., while with isolcpus we
> expect it to be (0xFFFF - isolcpus).
> > This breaks the example from above, as now the non-RT thread can also be
> scheduled on the RT CPU.
> >
> > When only using cgroup-v2, we can isolate our RT process by placing it in a
> cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
> > However, we do not know of a strategy to set a default affinity:
> > Given the example above, we have no way to ensure that newly created
> threads are born with an affinity of just 0x2 (without changing the application).
> >
> > Finally, isolcpus itself is deprecated since kernel 5.4.
> 
> Where is this the deprecation of isolcpus announced/ written?

https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
isolcpus=       [KNL,SMP,ISOL] Isolate a given set of CPUs from disturbance.
                        [Deprecated - use cpusets instead]
                        Format: [flag-list,]<cpu-list>

> 
> > Questions:
> >
> > 1. What is the best strategy to "isolcpus" similar semantics with cgroups-v2?
> > 2. Is there a way to specify the default affinity (within a cgroup)
> >
> > We are currently at a point where we would write patches to add a default
> affinity feature to cpusets of cgroupv2.
> > But maybe that is not needed or would be the wrong direction, so we wanted
> to discuss first.
> >
> > Best regards,
> > Felix Mößbauer
> > Siemens AG
> 
> Sebastian

Best regards,
Felix

Re: Questions about replacing isolcpus by cgroup-v2

[Jan Kiszka]

On 12.11.21 16:45, Moessbauer, Felix (T RDA IOT SES-DE) wrote:
> Hi Sebastian,
> 
>> -----Original Message-----
>> From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
>> Sent: Friday, November 12, 2021 4:37 PM
>> To: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@siemens.com>;
>> cgroups@vger.kernel.org
>> Cc: linux-rt-users@vger.kernel.org; Schild, Henning (T RDA IOT SES-DE)
>> <henning.schild@siemens.com>; Kiszka, Jan (T RDA IOT)
>> <jan.kiszka@siemens.com>; Schmidt, Adriaan (T RDA IOT SES-DE)
>> <adriaan.schmidt@siemens.com>; Frederic Weisbecker <frederic@kernel.org>
>> Subject: Re: Questions about replacing isolcpus by cgroup-v2
>>
>> On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:
>>> Dear subscribers,
>> Hi,
>>
>> I Cced cgroups@vger since thus question fits there better.
>> I Cced Frederic in case he has come clues regarding isolcpus and cgroups.
> 
> Indeed. Thanks!
> 
>>
>>> we are currently evaluating how to rework realtime tuning to use cgroup-v2
>> cpusets instead of the isolcpus kernel parameter.
>>> Our use-case are realtime applications with rt and non-rt threads. Hereby, the
>> non-rt thread might create additional non-rt threads:
>>>
>>> Example (RT CPU=1, 4 CPUs):
>>> - Non-RT Thread (A) with default affinity 0xD (1101b)
>>> - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
>>>
>>> When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly
>> works:
>>> Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a
>> default affinity of 0xD.
>>> By that, independent of the threads' priorities, we can ensure that nothing is
>> scheduled on our RT cpu (except from kernel threads, etc...).
>>>
>>> During this journey, we discovered the following:
>>>
>>> Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
>>> When activating the cpuset controller on a cgroup (for the first time), all
>> default CPU affinities are reset.
>>> By that, also the default affinity is set to 0xFFFF..., while with isolcpus we
>> expect it to be (0xFFFF - isolcpus).
>>> This breaks the example from above, as now the non-RT thread can also be
>> scheduled on the RT CPU.
>>>
>>> When only using cgroup-v2, we can isolate our RT process by placing it in a
>> cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
>>> However, we do not know of a strategy to set a default affinity:
>>> Given the example above, we have no way to ensure that newly created
>> threads are born with an affinity of just 0x2 (without changing the application).
>>>
>>> Finally, isolcpus itself is deprecated since kernel 5.4.
>>
>> Where is this the deprecation of isolcpus announced/ written?
> 
> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
> isolcpus=       [KNL,SMP,ISOL] Isolate a given set of CPUs from disturbance.
>                         [Deprecated - use cpusets instead]
>                         Format: [flag-list,]<cpu-list>
> 

That was Frederic himself via b0d40d2b22fe - but already for 4.15...

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Questions about replacing isolcpus by cgroup-v2

[Sebastian Andrzej Siewior]

On 2021-11-12 16:46:08 [+0100], Jan Kiszka wrote:
> > https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
> > isolcpus=       [KNL,SMP,ISOL] Isolate a given set of CPUs from disturbance.
> >                         [Deprecated - use cpusets instead]
> >                         Format: [flag-list,]<cpu-list>
> > 
> 
> That was Frederic himself via b0d40d2b22fe - but already for 4.15...

Just found out that myself after getting the pointer from Felix. Maybe
he is able to answer the questions here :)

> Jan
> 

Sebastian

Re: Questions about replacing isolcpus by cgroup-v2

[Frederic Weisbecker]

On Fri, Nov 12, 2021 at 04:36:56PM +0100, Sebastian Andrzej Siewior wrote:
> On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:
> > Dear subscribers,
> Hi,
> 
> I Cced cgroups@vger since thus question fits there better.
> I Cced Frederic in case he has come clues regarding isolcpus and
> cgroups.
> 
> > we are currently evaluating how to rework realtime tuning to use cgroup-v2 cpusets instead of the isolcpus kernel parameter.
> > Our use-case are realtime applications with rt and non-rt threads. Hereby, the non-rt thread might create additional non-rt threads:
> > 
> > Example (RT CPU=1, 4 CPUs):
> > - Non-RT Thread (A) with default affinity 0xD (1101b)
> > - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
> > 
> > When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly works:
> > Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a default affinity of 0xD.
> > By that, independent of the threads' priorities, we can ensure that nothing is scheduled on our RT cpu (except from kernel threads, etc...).
> > 
> > During this journey, we discovered the following:
> > 
> > Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
> > When activating the cpuset controller on a cgroup (for the first time), all default CPU affinities are reset.
> > By that, also the default affinity is set to 0xFFFF..., while with isolcpus we expect it to be (0xFFFF - isolcpus).
> > This breaks the example from above, as now the non-RT thread can also be
> > scheduled on the RT CPU.

That sounds buggy from the cpuset-v2 side (adding the maintainers in Cc).

Also please have a look into "[PATCH v8 0/6] cgroup/cpuset: Add new cpuset
partition type & empty effecitve cpus":

	  https://lore.kernel.org/lkml/20211018143619.205065-1-longman@redhat.com/

This stuff adds support for a new "isolated" partition type on cpuset/cgroup-v2
which should behave just like isolcpus.

> > 
> > When only using cgroup-v2, we can isolate our RT process by placing it in a cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
> > However, we do not know of a strategy to set a default affinity:
> > Given the example above, we have no way to ensure that newly created threads are born with an affinity of just 0x2 (without changing the application).
> > 
> > Finally, isolcpus itself is deprecated since kernel 5.4.
> 
> Where is this the deprecation of isolcpus announced/ written?

We tried to deprecate it but too many people are still using it. Better pick an
interface that allows you to change the isolated set at runtime like
cpuset.sched_load_balance on cpuset/cgroup-v1 or the above patchset on v2.

Thanks.

Re: Questions about replacing isolcpus by cgroup-v2

[Henning Schild]

Am Fri, 12 Nov 2021 17:37:07 +0100
schrieb Frederic Weisbecker <frederic@kernel.org>:

> On Fri, Nov 12, 2021 at 04:36:56PM +0100, Sebastian Andrzej Siewior
> wrote:
> > On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:  
> > > Dear subscribers,  
> > Hi,
> > 
> > I Cced cgroups@vger since thus question fits there better.
> > I Cced Frederic in case he has come clues regarding isolcpus and
> > cgroups.
> >   
> > > we are currently evaluating how to rework realtime tuning to use
> > > cgroup-v2 cpusets instead of the isolcpus kernel parameter. Our
> > > use-case are realtime applications with rt and non-rt threads.
> > > Hereby, the non-rt thread might create additional non-rt threads:
> > > 
> > > Example (RT CPU=1, 4 CPUs):
> > > - Non-RT Thread (A) with default affinity 0xD (1101b)
> > > - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
> > > 
> > > When using pure isolcpus and cgroup-v1, just setting isolcpus=1
> > > perfectly works: Thread A gets affinity 0xD, Thread B gets 0x2
> > > and additional threads get a default affinity of 0xD. By that,
> > > independent of the threads' priorities, we can ensure that
> > > nothing is scheduled on our RT cpu (except from kernel threads,
> > > etc...).
> > > 
> > > During this journey, we discovered the following:
> > > 
> > > Using cgroup-v2 cpusets and isolcpus together seems to be
> > > incompatible: When activating the cpuset controller on a cgroup
> > > (for the first time), all default CPU affinities are reset. By
> > > that, also the default affinity is set to 0xFFFF..., while with
> > > isolcpus we expect it to be (0xFFFF - isolcpus). This breaks the
> > > example from above, as now the non-RT thread can also be
> > > scheduled on the RT CPU.  
> 
> That sounds buggy from the cpuset-v2 side (adding the maintainers in
> Cc).

We thought the same but ended up saying that it might just be related
to the deprecation. But regarding that as a bug could help fix it in
stable kernels and distros. Because the patches will need some time and
will likely not get backported.

Henning

> Also please have a look into "[PATCH v8 0/6] cgroup/cpuset: Add new
> cpuset partition type & empty effecitve cpus":
> 
> 	  https://lore.kernel.org/lkml/20211018143619.205065-1-longman@redhat.com/
> 
> This stuff adds support for a new "isolated" partition type on
> cpuset/cgroup-v2 which should behave just like isolcpus.
> 
> > > 
> > > When only using cgroup-v2, we can isolate our RT process by
> > > placing it in a cgroup with CPUs=0,1 and remove CPU=1 from all
> > > other cgroups. However, we do not know of a strategy to set a
> > > default affinity: Given the example above, we have no way to
> > > ensure that newly created threads are born with an affinity of
> > > just 0x2 (without changing the application).
> > > 
> > > Finally, isolcpus itself is deprecated since kernel 5.4.  
> > 
> > Where is this the deprecation of isolcpus announced/ written?  
> 
> We tried to deprecate it but too many people are still using it.
> Better pick an interface that allows you to change the isolated set
> at runtime like cpuset.sched_load_balance on cpuset/cgroup-v1 or the
> above patchset on v2.
> 
> Thanks.

RE: Questions about replacing isolcpus by cgroup-v2

[Moessbauer, Felix]

Hi Sebastian,

> -----Original Message-----
> From: Frederic Weisbecker <frederic@kernel.org>
> Sent: Friday, November 12, 2021 5:37 PM
> To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Cc: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@siemens.com>;
> cgroups@vger.kernel.org; linux-rt-users@vger.kernel.org; Schild, Henning (T
> RDA IOT SES-DE) <henning.schild@siemens.com>; Kiszka, Jan (T RDA IOT)
> <jan.kiszka@siemens.com>; Schmidt, Adriaan (T RDA IOT SES-DE)
> <adriaan.schmidt@siemens.com>; Zefan Li <lizefan.x@bytedance.com>; Tejun
> Heo <tj@kernel.org>; Johannes Weiner <hannes@cmpxchg.org>; Waiman Long
> <longman@redhat.com>
> Subject: Re: Questions about replacing isolcpus by cgroup-v2
> 
> On Fri, Nov 12, 2021 at 04:36:56PM +0100, Sebastian Andrzej Siewior wrote:
> > On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:
> > > Dear subscribers,
> > Hi,
> >
> > I Cced cgroups@vger since thus question fits there better.
> > I Cced Frederic in case he has come clues regarding isolcpus and
> > cgroups.
> >
> > > we are currently evaluating how to rework realtime tuning to use cgroup-v2
> cpusets instead of the isolcpus kernel parameter.
> > > Our use-case are realtime applications with rt and non-rt threads. Hereby,
> the non-rt thread might create additional non-rt threads:
> > >
> > > Example (RT CPU=1, 4 CPUs):
> > > - Non-RT Thread (A) with default affinity 0xD (1101b)
> > > - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)
> > >
> > > When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly
> works:
> > > Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a
> default affinity of 0xD.
> > > By that, independent of the threads' priorities, we can ensure that nothing is
> scheduled on our RT cpu (except from kernel threads, etc...).
> > >
> > > During this journey, we discovered the following:
> > >
> > > Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
> > > When activating the cpuset controller on a cgroup (for the first time), all
> default CPU affinities are reset.
> > > By that, also the default affinity is set to 0xFFFF..., while with isolcpus we
> expect it to be (0xFFFF - isolcpus).
> > > This breaks the example from above, as now the non-RT thread can
> > > also be scheduled on the RT CPU.
> 
> That sounds buggy from the cpuset-v2 side (adding the maintainers in Cc).
> 
> Also please have a look into "[PATCH v8 0/6] cgroup/cpuset: Add new cpuset
> partition type & empty effecitve cpus":
> 
> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kern
> el.org%2Flkml%2F20211018143619.205065-1-
> longman%40redhat.com%2F&amp;data=04%7C01%7Cfelix.moessbauer%40sie
> mens.com%7C1a74cbf4e3d140a9031808d9a5faad3c%7C38ae3bcd95794fd4add
> ab42e1495d55a%7C1%7C0%7C637723318334809165%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
> n0%3D%7C1000&amp;sdata=Jm3j2vDvOOtikU5ZpusupQ6d6koPII9oYZZhpDUkvY
> c%3D&amp;reserved=0
> 
> This stuff adds support for a new "isolated" partition type on cpuset/cgroup-v2
> which should behave just like isolcpus.

I already tested the patch and reported back on the ML.
However, it only covers load-balancing aspects, not isolcpu like default affinities.
When setting cpusets.cpus.partition=isolated, you get similar behavior as with =root
(cpus are removed from all other groups), but the schedulers load-balancing is disabled
for this domain.
For details, please have a look in the other thread. 

> 
> > >
> > > When only using cgroup-v2, we can isolate our RT process by placing it in a
> cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.
> > > However, we do not know of a strategy to set a default affinity:
> > > Given the example above, we have no way to ensure that newly created
> threads are born with an affinity of just 0x2 (without changing the application).
> > >
> > > Finally, isolcpus itself is deprecated since kernel 5.4.
> >
> > Where is this the deprecation of isolcpus announced/ written?
> 
> We tried to deprecate it but too many people are still using it. Better pick an
> interface that allows you to change the isolated set at runtime like
> cpuset.sched_load_balance on cpuset/cgroup-v1 or the above patchset on v2.
> 

Currently, the only workaround we know of to get isolcpu semantics on systems where
other tools like container runtimes or libvirt fiddle around with the cpuset controller, is to simply enforce the cgroups-v1.
But maybe we are just running into the bug from above.

Felix

> Thanks.