Question regarding sched_setaffinity

Question regarding sched_setaffinity

[Karthik Singaram Lakshmanan]

Hi,
 I am a noob to SMP real-time scheduling in linux, I have a question of
sched_setaffinity(). If I restrict an application to a single CPU
using the appropriate cpumask for sched_setaffinity(), Is it
guaranteed that the kernel will never schedule it on any other CPU? I
am asking this because looking at kernel/sched.c seems to use
cpu_clear during load balancing. I am interested in knowing whether the
cache will be invalidated when my task is assigned to a single CPU.
Are there exceptions where the cpu-mask will be overridden?
Thanks for taking the time to answer my query.

Thanks,
Karthik

Re: Question regarding sched_setaffinity

[Steven Rostedt]

On Sun, 31 Aug 2008, Karthik Singaram Lakshmanan wrote:

> Hi,
>  I am a noob to SMP real-time scheduling in linux, I have a question of
> sched_setaffinity(). If I restrict an application to a single CPU
> using the appropriate cpumask for sched_setaffinity(), Is it
> guaranteed that the kernel will never schedule it on any other CPU? I
> am asking this because looking at kernel/sched.c seems to use
> cpu_clear during load balancing. I am interested in knowing whether the
> cache will be invalidated when my task is assigned to a single CPU.
> Are there exceptions where the cpu-mask will be overridden?
> Thanks for taking the time to answer my query.

A task should never be scheduled on a CPU that is not in its affinity. 
(although I hear a rumor that if a task is bound to a single CPU, and that 
CPU is taken offline, it will be migrated. But I doubt this is true, since 
there are tasks that would crash the system if this were true).

-- Steve

Re: Question regarding sched_setaffinity

[Mark Hounschell]

Steven Rostedt wrote:
> On Sun, 31 Aug 2008, Karthik Singaram Lakshmanan wrote:
> 
>> Hi,
>>  I am a noob to SMP real-time scheduling in linux, I have a question of
>> sched_setaffinity(). If I restrict an application to a single CPU
>> using the appropriate cpumask for sched_setaffinity(), Is it
>> guaranteed that the kernel will never schedule it on any other CPU? I
>> am asking this because looking at kernel/sched.c seems to use
>> cpu_clear during load balancing. I am interested in knowing whether the
>> cache will be invalidated when my task is assigned to a single CPU.
>> Are there exceptions where the cpu-mask will be overridden?
>> Thanks for taking the time to answer my query.
> 
> A task should never be scheduled on a CPU that is not in its affinity. 
> (although I hear a rumor that if a task is bound to a single CPU, and that 
> CPU is taken offline, it will be migrated. But I doubt this is true, since 
> there are tasks that would crash the system if this were true).
> 
> -- Steve
> 

It seems to be true. It's affinity mask gets changed to the CPUs left online.
At least according to sched_getaffinity.

Mark

Re: Question regarding sched_setaffinity

[Steven Rostedt]

On Fri, 5 Sep 2008, Mark Hounschell wrote:

> Steven Rostedt wrote:
> > On Sun, 31 Aug 2008, Karthik Singaram Lakshmanan wrote:
> > 
> > > Hi,
> > >  I am a noob to SMP real-time scheduling in linux, I have a question of
> > > sched_setaffinity(). If I restrict an application to a single CPU
> > > using the appropriate cpumask for sched_setaffinity(), Is it
> > > guaranteed that the kernel will never schedule it on any other CPU? I
> > > am asking this because looking at kernel/sched.c seems to use
> > > cpu_clear during load balancing. I am interested in knowing whether the
> > > cache will be invalidated when my task is assigned to a single CPU.
> > > Are there exceptions where the cpu-mask will be overridden?
> > > Thanks for taking the time to answer my query.
> > 
> > A task should never be scheduled on a CPU that is not in its affinity.
> > (although I hear a rumor that if a task is bound to a single CPU, and that
> > CPU is taken offline, it will be migrated. But I doubt this is true, since
> > there are tasks that would crash the system if this were true).
> > 
> > -- Steve
> > 
> 
> It seems to be true. It's affinity mask gets changed to the CPUs left online.
> At least according to sched_getaffinity.


Hmm, it is true:

[root@bxrhel51 bin]# ps ax |grep yes
 6560 pts/1    R+     0:31 yes
 6566 pts/0    S+     0:00 grep yes
[root@bxrhel51 bin]# taskset -p 8 6560
pid 6560's current affinity mask: f
pid 6560's new affinity mask: 8
[root@bxrhel51 bin]# taskset -p 6560
pid 6560's current affinity mask: 8
[root@bxrhel51 bin]# echo 0 > /sys/devices/system/cpu/cpu3/online
[root@bxrhel51 bin]# taskset -p 6560
pid 6560's current affinity mask: 7


I ran "yes" and made the affinity set to CPU3. Then I took CPU3 offline 
and this just made the task move to _all_ other online CPUS.

-- Steve

Re: Question regarding sched_setaffinity

[Steven Rostedt]

On Fri, 5 Sep 2008, Mark Hounschell wrote:

> Steven Rostedt wrote:
> > On Sun, 31 Aug 2008, Karthik Singaram Lakshmanan wrote:
> > 
> > > Hi,
> > >  I am a noob to SMP real-time scheduling in linux, I have a question of
> > > sched_setaffinity(). If I restrict an application to a single CPU
> > > using the appropriate cpumask for sched_setaffinity(), Is it
> > > guaranteed that the kernel will never schedule it on any other CPU? I
> > > am asking this because looking at kernel/sched.c seems to use
> > > cpu_clear during load balancing. I am interested in knowing whether the
> > > cache will be invalidated when my task is assigned to a single CPU.
> > > Are there exceptions where the cpu-mask will be overridden?
> > > Thanks for taking the time to answer my query.
> > 
> > A task should never be scheduled on a CPU that is not in its affinity.
> > (although I hear a rumor that if a task is bound to a single CPU, and that
> > CPU is taken offline, it will be migrated. But I doubt this is true, since
> > there are tasks that would crash the system if this were true).
> > 
> > -- Steve
> > 
> 
> It seems to be true. It's affinity mask gets changed to the CPUs left online.
> At least according to sched_getaffinity.


Note, I'm not just talking about that affinity being masked off, I'm 
talking about a task that is bounded to only that cpu that is about to go 
offline.

If you have a mask (bit mask) 0xf to be on all 4 cpus, and cpu 3 goes 
offline, all those tasks will now be 0x7. But if you started with a task 
with a affinity of mask 0x8, then when it goes offline, it too will be 
0x7. The difference here is that this task is now running on CPUS that it 
wasn't able to run on before going down.

At least there's a warning about it in dmesg:

"process 6560 (yes) no longer affine to cpu3"

-- Steve

Re: Question regarding sched_setaffinity

[Mark Hounschell]

Steven Rostedt wrote:
> On Fri, 5 Sep 2008, Mark Hounschell wrote:
> 
>> Steven Rostedt wrote:
>>> On Sun, 31 Aug 2008, Karthik Singaram Lakshmanan wrote:
>>>
>>>> Hi,
>>>>  I am a noob to SMP real-time scheduling in linux, I have a question of
>>>> sched_setaffinity(). If I restrict an application to a single CPU
>>>> using the appropriate cpumask for sched_setaffinity(), Is it
>>>> guaranteed that the kernel will never schedule it on any other CPU? I
>>>> am asking this because looking at kernel/sched.c seems to use
>>>> cpu_clear during load balancing. I am interested in knowing whether the
>>>> cache will be invalidated when my task is assigned to a single CPU.
>>>> Are there exceptions where the cpu-mask will be overridden?
>>>> Thanks for taking the time to answer my query.
>>> A task should never be scheduled on a CPU that is not in its affinity.
>>> (although I hear a rumor that if a task is bound to a single CPU, and that
>>> CPU is taken offline, it will be migrated. But I doubt this is true, since
>>> there are tasks that would crash the system if this were true).
>>>
>>> -- Steve
>>>
>> It seems to be true. It's affinity mask gets changed to the CPUs left online.
>> At least according to sched_getaffinity.
> 
> 
> Note, I'm not just talking about that affinity being masked off, I'm 
> talking about a task that is bounded to only that cpu that is about to go 
> offline.
> 
> If you have a mask (bit mask) 0xf to be on all 4 cpus, and cpu 3 goes 
> offline, all those tasks will now be 0x7. But if you started with a task 
> with a affinity of mask 0x8, then when it goes offline, it too will be 
> 0x7. The difference here is that this task is now running on CPUS that it 
> wasn't able to run on before going down.
> 
> At least there's a warning about it in dmesg:
> 
> "process 6560 (yes) no longer affine to cpu3"
> 
> -- Steve
> 
> 

Right, thats what I was referring to also. A task bound to a single cpu
all by its self.

Mark

Re: Question regarding sched_setaffinity

[Karthik Singaram Lakshmanan]

Thanks for all your prompt replies. I really appreciate your help. I
was able to build some test programs to verify the behavior, I wanted
to make sure that I am not missing any corner cases. In my target
scenario, we do not consider taking CPUs offline, therefore
sched_setaffinity works perfectly fine.

Thanks,
Karthik