Question Regarding isolcpus

Question Regarding isolcpus

[Joseph Salisbury]

Hi All,

I have a question regarding the isolcpus parameter.  I've been seeing 
this parameter commonly used. However, in the kernel.org 
documentation[0], isolcpus is listed as depreciated.

Is it the case that isolcpus should not be used at all?  I've seen it 
used in conjunction with taskset.  However, should we now be telling rt 
users to use only cpusets in cgroups?  I see that CPUAffinity can be set 
in /etc/systemd/system.conf.  Is that the preferred method, so the 
process scheduler will automatically migrate processes between the 
cpusets in the cgroup cpuset or the list set by CPUAffinity?

I'm sure there are some presentations or papers on this, so I'll dig 
around as well :-)

Thanks,

Joe


[0] 
https://www.kernel.org/doc/html/v5.15/admin-guide/kernel-parameters.html#:~:text=Deprecated%20%2D%20use%20cpusets%20instead 

Re: Question Regarding isolcpus

[Sebastian Andrzej Siewior]

On 2023-09-26 12:45:14 [-0400], Joseph Salisbury wrote:
> Hi All,
Hi,

> I have a question regarding the isolcpus parameter.  I've been seeing this
> parameter commonly used. However, in the kernel.org documentation[0],
> isolcpus is listed as depreciated.
> 
> Is it the case that isolcpus should not be used at all?  I've seen it used
> in conjunction with taskset.  However, should we now be telling rt users to
> use only cpusets in cgroups?  I see that CPUAffinity can be set in
> /etc/systemd/system.conf.  Is that the preferred method, so the process
> scheduler will automatically migrate processes between the cpusets in the
> cgroup cpuset or the list set by CPUAffinity?

Frederic might know if there is an actual timeline to remove it. The
suggestions since then is to use cpusets which should be more flexible.
There was also some work (which went into v6.1 I think) to be able to
reconfigure the partitions at run-time while isolcpus= is a boot time
option.
From what I remember, you have a default/system cpuset which all tasks
use by default and then you can add another cpuset for the "isolated"
CPUs. Based on the partition it can be either the default one or
isolated [0]. The latter would exclude the CPUs from load balancing
which is what isolcpus= does.

[0] f28e22441f353 ("cgroup/cpuset: Add a new isolated cpus.partition type")

> Thanks,
> 
> Joe

Sebastian

Re: Question Regarding isolcpus

[Rod Webster]

We in the linuxCNC community would not be in favour of removing isolcpus.
In our application there are only one or sometimes two RT threads, a
large user base deployed on a variety of platforms.
isolcpus gives our users a simple way to optionally improve RT performance.
When I review the cpusets documentation, it adds additional complexity
to configuration.
Our users are machinists, not IT professionals so would likely
struggle to configure a cpusets environment.
The LinuxCNC application is deployed on machine controllers so boot
time parameters are not an issue and likely preferred.

It would be appreciated if you could consult more widely before
removing isolcpus as it will affect our users
scattered through every country in the world.


Rod Webster
1300 896 832
+61 435 765 611
VMN®
www.vmn.com.au

Rod Webster
1300 896 832
+61 435 765 611
VMN®
www.vmn.com.au

Sole Queensland Distributor


On Thu, 28 Sept 2023 at 18:39, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> On 2023-09-26 12:45:14 [-0400], Joseph Salisbury wrote:
> > Hi All,
> Hi,
>
> > I have a question regarding the isolcpus parameter.  I've been seeing this
> > parameter commonly used. However, in the kernel.org documentation[0],
> > isolcpus is listed as depreciated.
> >
> > Is it the case that isolcpus should not be used at all?  I've seen it used
> > in conjunction with taskset.  However, should we now be telling rt users to
> > use only cpusets in cgroups?  I see that CPUAffinity can be set in
> > /etc/systemd/system.conf.  Is that the preferred method, so the process
> > scheduler will automatically migrate processes between the cpusets in the
> > cgroup cpuset or the list set by CPUAffinity?
>
> Frederic might know if there is an actual timeline to remove it. The
> suggestions since then is to use cpusets which should be more flexible.
> There was also some work (which went into v6.1 I think) to be able to
> reconfigure the partitions at run-time while isolcpus= is a boot time
> option.
> From what I remember, you have a default/system cpuset which all tasks
> use by default and then you can add another cpuset for the "isolated"
> CPUs. Based on the partition it can be either the default one or
> isolated [0]. The latter would exclude the CPUs from load balancing
> which is what isolcpus= does.
>
> [0] f28e22441f353 ("cgroup/cpuset: Add a new isolated cpus.partition type")
>
> > Thanks,
> >
> > Joe
>
> Sebastian
>

Re: Question Regarding isolcpus

[Gautham]

+1
There are several applications ( high throughput, simple control
systems ) where isolcpus is used as a simple way to extract maximum
performance.

Request the team to please consult more widely before removing isolcpu.
Please let us know if a different venue/mailing list is the best site
for feedback regarding this topic.

Best
Gautham


On Thu, 28 Sept 2023 at 15:20, Rod Webster <rod@vmn.com.au> wrote:
>
> We in the linuxCNC community would not be in favour of removing isolcpus.
> In our application there are only one or sometimes two RT threads, a
> large user base deployed on a variety of platforms.
> isolcpus gives our users a simple way to optionally improve RT performance.
> When I review the cpusets documentation, it adds additional complexity
> to configuration.
> Our users are machinists, not IT professionals so would likely
> struggle to configure a cpusets environment.
> The LinuxCNC application is deployed on machine controllers so boot
> time parameters are not an issue and likely preferred.
>
> It would be appreciated if you could consult more widely before
> removing isolcpus as it will affect our users
> scattered through every country in the world.
>
>
> Rod Webster
> 1300 896 832
> +61 435 765 611
> VMN®
> www.vmn.com.au
>
> Rod Webster
> 1300 896 832
> +61 435 765 611
> VMN®
> www.vmn.com.au
>
> Sole Queensland Distributor
>
>
> On Thu, 28 Sept 2023 at 18:39, Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> wrote:
> >
> > On 2023-09-26 12:45:14 [-0400], Joseph Salisbury wrote:
> > > Hi All,
> > Hi,
> >
> > > I have a question regarding the isolcpus parameter.  I've been seeing this
> > > parameter commonly used. However, in the kernel.org documentation[0],
> > > isolcpus is listed as depreciated.
> > >
> > > Is it the case that isolcpus should not be used at all?  I've seen it used
> > > in conjunction with taskset.  However, should we now be telling rt users to
> > > use only cpusets in cgroups?  I see that CPUAffinity can be set in
> > > /etc/systemd/system.conf.  Is that the preferred method, so the process
> > > scheduler will automatically migrate processes between the cpusets in the
> > > cgroup cpuset or the list set by CPUAffinity?
> >
> > Frederic might know if there is an actual timeline to remove it. The
> > suggestions since then is to use cpusets which should be more flexible.
> > There was also some work (which went into v6.1 I think) to be able to
> > reconfigure the partitions at run-time while isolcpus= is a boot time
> > option.
> > From what I remember, you have a default/system cpuset which all tasks
> > use by default and then you can add another cpuset for the "isolated"
> > CPUs. Based on the partition it can be either the default one or
> > isolated [0]. The latter would exclude the CPUs from load balancing
> > which is what isolcpus= does.
> >
> > [0] f28e22441f353 ("cgroup/cpuset: Add a new isolated cpus.partition type")
> >
> > > Thanks,
> > >
> > > Joe
> >
> > Sebastian
> >
>


-- 

Gautham Ponnu |  http://gauthamponnu.com

Re: Question Regarding isolcpus

[Joseph Salisbury]

On 9/28/23 04:39, Sebastian Andrzej Siewior wrote:
> On 2023-09-26 12:45:14 [-0400], Joseph Salisbury wrote:
>> Hi All,
> Hi,
>
>> I have a question regarding the isolcpus parameter.  I've been seeing this
>> parameter commonly used. However, in the kernel.org documentation[0],
>> isolcpus is listed as depreciated.
>>
>> Is it the case that isolcpus should not be used at all?  I've seen it used
>> in conjunction with taskset.  However, should we now be telling rt users to
>> use only cpusets in cgroups?  I see that CPUAffinity can be set in
>> /etc/systemd/system.conf.  Is that the preferred method, so the process
>> scheduler will automatically migrate processes between the cpusets in the
>> cgroup cpuset or the list set by CPUAffinity?
> Frederic might know if there is an actual timeline to remove it. The
> suggestions since then is to use cpusets which should be more flexible.
> There was also some work (which went into v6.1 I think) to be able to
> reconfigure the partitions at run-time while isolcpus= is a boot time
> option.
>  From what I remember, you have a default/system cpuset which all tasks
> use by default and then you can add another cpuset for the "isolated"
> CPUs. Based on the partition it can be either the default one or
> isolated [0]. The latter would exclude the CPUs from load balancing
> which is what isolcpus= does.
>
> [0] f28e22441f353 ("cgroup/cpuset: Add a new isolated cpus.partition type")

This question may be for the cgroups folks.  The kernel.org 
documentation has a WARNING which states: "cgroup2 doesn't yet support 
control of realtime processes and the cpu controller can only be enabled 
when all RT processes are in the root cgroup "[0]. Does this mean 
real-time processes are only supported on cgroupsV1?

Also, this warning is stated for the "CPU" Controller, but there is no 
mention of this for a "cpuset" controller. Does this imply that 
real-time processes are supported with "cpuset" controllers?


Thanks,

Joe

[0] 
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#controllers

>
>> Thanks,
>>
>> Joe
> Sebastian
>

Re: Question Regarding isolcpus

[Waiman Long]

On 10/12/23 13:27, Joseph Salisbury wrote:
>
>
> On 9/28/23 04:39, Sebastian Andrzej Siewior wrote:
>> On 2023-09-26 12:45:14 [-0400], Joseph Salisbury wrote:
>>> Hi All,
>> Hi,
>>
>>> I have a question regarding the isolcpus parameter.  I've been 
>>> seeing this
>>> parameter commonly used. However, in the kernel.org documentation[0],
>>> isolcpus is listed as depreciated.
>>>
>>> Is it the case that isolcpus should not be used at all?  I've seen 
>>> it used
>>> in conjunction with taskset.  However, should we now be telling rt 
>>> users to
>>> use only cpusets in cgroups?  I see that CPUAffinity can be set in
>>> /etc/systemd/system.conf.  Is that the preferred method, so the process
>>> scheduler will automatically migrate processes between the cpusets 
>>> in the
>>> cgroup cpuset or the list set by CPUAffinity?
>> Frederic might know if there is an actual timeline to remove it. The
>> suggestions since then is to use cpusets which should be more flexible.
>> There was also some work (which went into v6.1 I think) to be able to
>> reconfigure the partitions at run-time while isolcpus= is a boot time
>> option.
>>  From what I remember, you have a default/system cpuset which all tasks
>> use by default and then you can add another cpuset for the "isolated"
>> CPUs. Based on the partition it can be either the default one or
>> isolated [0]. The latter would exclude the CPUs from load balancing
>> which is what isolcpus= does.
>>
>> [0] f28e22441f353 ("cgroup/cpuset: Add a new isolated cpus.partition 
>> type")
>
> This question may be for the cgroups folks.  The kernel.org 
> documentation has a WARNING which states: "cgroup2 doesn't yet support 
> control of realtime processes and the cpu controller can only be 
> enabled when all RT processes are in the root cgroup "[0]. Does this 
> mean real-time processes are only supported on cgroupsV1?
>
> Also, this warning is stated for the "CPU" Controller, but there is no 
> mention of this for a "cpuset" controller. Does this imply that 
> real-time processes are supported with "cpuset" controllers?

Yes, the quoted description applies only to cpu controller. Even for v1 
cpu controller, the realtime support is problematic and there is no easy 
solution to that. That is why cgroup v2 doesn't support it.

For other controllers, whether the processes are RT or not are 
irrelevant. They are equally supported.

Cheers,
Longman

Re: Question Regarding isolcpus

[Joseph Salisbury]

On 10/12/23 15:10, Waiman Long wrote:
> On 10/12/23 13:27, Joseph Salisbury wrote:
>>
>>
>> On 9/28/23 04:39, Sebastian Andrzej Siewior wrote:
>>> On 2023-09-26 12:45:14 [-0400], Joseph Salisbury wrote:
>>>> Hi All,
>>> Hi,
>>>
>>>> I have a question regarding the isolcpus parameter.  I've been 
>>>> seeing this
>>>> parameter commonly used. However, in the kernel.org documentation[0],
>>>> isolcpus is listed as depreciated.
>>>>
>>>> Is it the case that isolcpus should not be used at all? I've seen 
>>>> it used
>>>> in conjunction with taskset.  However, should we now be telling rt 
>>>> users to
>>>> use only cpusets in cgroups?  I see that CPUAffinity can be set in
>>>> /etc/systemd/system.conf.  Is that the preferred method, so the 
>>>> process
>>>> scheduler will automatically migrate processes between the cpusets 
>>>> in the
>>>> cgroup cpuset or the list set by CPUAffinity?
>>> Frederic might know if there is an actual timeline to remove it. The
>>> suggestions since then is to use cpusets which should be more flexible.
>>> There was also some work (which went into v6.1 I think) to be able to
>>> reconfigure the partitions at run-time while isolcpus= is a boot time
>>> option.
>>>  From what I remember, you have a default/system cpuset which all tasks
>>> use by default and then you can add another cpuset for the "isolated"
>>> CPUs. Based on the partition it can be either the default one or
>>> isolated [0]. The latter would exclude the CPUs from load balancing
>>> which is what isolcpus= does.
>>>
>>> [0] f28e22441f353 ("cgroup/cpuset: Add a new isolated cpus.partition 
>>> type")
>>
>> This question may be for the cgroups folks.  The kernel.org 
>> documentation has a WARNING which states: "cgroup2 doesn't yet 
>> support control of realtime processes and the cpu controller can only 
>> be enabled when all RT processes are in the root cgroup "[0]. Does 
>> this mean real-time processes are only supported on cgroupsV1?
>>
>> Also, this warning is stated for the "CPU" Controller, but there is 
>> no mention of this for a "cpuset" controller. Does this imply that 
>> real-time processes are supported with "cpuset" controllers?
>
> Yes, the quoted description applies only to cpu controller. Even for 
> v1 cpu controller, the realtime support is problematic and there is no 
> easy solution to that. That is why cgroup v2 doesn't support it.
>
> For other controllers, whether the processes are RT or not are 
> irrelevant. They are equally supported.
>
> Cheers,
> Longman
Thanks for the feedback, Longman!

>
>

Re: Question Regarding isolcpus

[Waiman Long]

On 10/12/23 15:23, Joseph Salisbury wrote:
>
>
> On 10/12/23 15:10, Waiman Long wrote:
>> On 10/12/23 13:27, Joseph Salisbury wrote:
>>>
>>>
>>> On 9/28/23 04:39, Sebastian Andrzej Siewior wrote:
>>>> On 2023-09-26 12:45:14 [-0400], Joseph Salisbury wrote:
>>>>> Hi All,
>>>> Hi,
>>>>
>>>>> I have a question regarding the isolcpus parameter.  I've been 
>>>>> seeing this
>>>>> parameter commonly used. However, in the kernel.org documentation[0],
>>>>> isolcpus is listed as depreciated.
>>>>>
>>>>> Is it the case that isolcpus should not be used at all? I've seen 
>>>>> it used
>>>>> in conjunction with taskset.  However, should we now be telling rt 
>>>>> users to
>>>>> use only cpusets in cgroups?  I see that CPUAffinity can be set in
>>>>> /etc/systemd/system.conf.  Is that the preferred method, so the 
>>>>> process
>>>>> scheduler will automatically migrate processes between the cpusets 
>>>>> in the
>>>>> cgroup cpuset or the list set by CPUAffinity?
>>>> Frederic might know if there is an actual timeline to remove it. The
>>>> suggestions since then is to use cpusets which should be more 
>>>> flexible.
>>>> There was also some work (which went into v6.1 I think) to be able to
>>>> reconfigure the partitions at run-time while isolcpus= is a boot time
>>>> option.
>>>>  From what I remember, you have a default/system cpuset which all 
>>>> tasks
>>>> use by default and then you can add another cpuset for the "isolated"
>>>> CPUs. Based on the partition it can be either the default one or
>>>> isolated [0]. The latter would exclude the CPUs from load balancing
>>>> which is what isolcpus= does.
>>>>
>>>> [0] f28e22441f353 ("cgroup/cpuset: Add a new isolated 
>>>> cpus.partition type")
>>>
>>> This question may be for the cgroups folks.  The kernel.org 
>>> documentation has a WARNING which states: "cgroup2 doesn't yet 
>>> support control of realtime processes and the cpu controller can 
>>> only be enabled when all RT processes are in the root cgroup "[0]. 
>>> Does this mean real-time processes are only supported on cgroupsV1?
>>>
>>> Also, this warning is stated for the "CPU" Controller, but there is 
>>> no mention of this for a "cpuset" controller. Does this imply that 
>>> real-time processes are supported with "cpuset" controllers?
>>
>> Yes, the quoted description applies only to cpu controller. Even for 
>> v1 cpu controller, the realtime support is problematic and there is 
>> no easy solution to that. That is why cgroup v2 doesn't support it.
>>
>> For other controllers, whether the processes are RT or not are 
>> irrelevant. They are equally supported.
>>
>> Cheers,
>> Longman
> Thanks for the feedback, Longman!
>
One further tidbit is the fact that the deadline scheduling policy can 
be used as a replacement of using cgroup v1 cpu controller RT knobs to 
place a limit one how many RT tasks can run on a CPU.

Cheers,
Longman