Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[RAKOTOSALAMA, Nirilanto]

Thanks Jan,

It's more and more clear.
Just a question, I've a little doubt. Native tasks do not inherit
affinity from their parent process. I've read that cpu affinity 
setting at pthread creation is only availablle in kernel space.
Do Posix skin threads inherit affinity from their parent, in user space?

If not, when I "sched_setaffinity" a parent process, I have to 
recursively "sched_setaffinity" all its threads and force them to
switch to primary mode. Is it right ?

> >>> Hello,
> >>>
> >>> I have some question about CPU affinity with Xenomai 2.3.1.
> >>> (it is to see if  xenomai support developing for a cpu 
> >> manager library is feasible)
> >>
> >> What will this library do? (/me just nosy)
> > 
> > In fact, there's simulation programs that run on many 
> different systems (hard + soft),
> > and to make this applications as portable as possible, they 
> use libraries developed for every
> > systems (dec alpha, pc, specific cards, linux, unix). And, 
> I search if xenomai could be used
> > with less re-implementation effort, so the more posix 
> function are developed by xenomai, 
> > the better it is, and easier it is (for me :p).
> > So first step is developing this libraries xenomai support 
> (e.g. timer, cpu).
> > This cpu library should provides "simple" functions like : 
> getting the number of cpu,
> > setting/getting the cpu affinity for a process passing its 
> pid, protect/un-protect cpu from running
> > unwanted process.
> 
> OK, I see.
> 
> > 
> >>> When using rt_task_create, CPU mask can be specified. 
> >> However, when sched_setaffinity is called 
> >>> to modify main program affinity, and after, rt_task_create 
> >> is called without specifying CPU mask,
> >>> did the child task inherits from the main program affinity ?
> >> If you don't provide a target CPU (either on creation or via
> >> /proc/xenomai/affinity), better consider the affinity as 
> unspecified
> >> (any possible CPU). The target CPU derives from Linux 
> >> decision where to
> >> put the thread non-RT boot code on, Xenomai just nails 
> that decision
> >> down until someone explicitly migrates.
> >>
> >>> Same question for Posix skin threads ?
> >>>
> >>> Can native task affinity be set other than when creating ? 
> >> using sdhed_setaffinity ?
> >>
> >> /proc/xenomai/affinity (has global scope) or 
> >> sched_setaffinity. Xenomai
> >> shadow threads follow the migration that may took place while in
> >> secondary mode, and sched_setaffinity comes with such a 
> mode switch.
> > 
> > Thanks, If I have well understood, tasks or threads 
> affinity can be set using /proc/xenomai/affinity
> > but, I have read that /proc/xenomai/affinity does not exist 
> in xenomai 2.3.1 and xenomai must be patched
> 
> Yes, it's an upcoming feature of 2.4.
> 
> > (I don't know how because I don't have nor SVN neither 
> internet connection on my xenomai boxes).
> 
> There is a daily snapshot on gna.org: 
> https://gna.org/svn/?group=xenomai
> 
> > So I can use sched_setaffinity, but it make switch to 
> secondary mode. 
> > Does'nt matter if only threads must be RT and 
> sched_setaffinity is called during a non-rt init phase, does it ?
> 
> Yep.
> 
> > That means that affinity does'nt depend on running on 
> xenomai or secondary mode ?
> 
> Sorry, doesn't parse for me right now.
> 
> > 
> > BTW, is protecting a CPU with xenomai possible ? 
> > Does xenomai use /dev/cpuset or an other device ?
> 
> Xenomai doesn't directly interact with cpuset -- but... if you force
> some RT process into a certain CPU and you don't override 
> this decision
> on RT thread creation, Xenomai will follow as explained.
> 
> Jan
> 
> 
> This mail has originated outside your organization, either 
> from an external partner or the Global Internet.
> Keep this in mind if you answer this message.
> 
> 
> 

This e-mail is intended only for the above addressee. It may contain privileged information.
If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. 
If you have received it in error please delete it and immediately notify the sender.
Security Notice: all e-mail, sent to or from this address, may be accessed by someone other than the recipient, for system management and security reasons. This access is controlled under Regulation of security reasons.
This access is controlled under Regulation of Investigatory Powers Act 2000, Lawful Business Practises.

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

RAKOTOSALAMA, Nirilanto wrote:
> Thanks Jan,
> 
> It's more and more clear.
> Just a question, I've a little doubt. Native tasks do not inherit
> affinity from their parent process. I've read that cpu affinity 
> setting at pthread creation is only availablle in kernel space.
> Do Posix skin threads inherit affinity from their parent, in user space?

IIRC, pthread CPU affinity is not (yet) covered by the POSIX specs, thus
the behaviour depends on Linux here. Moreover,
pthread_attr_setaffinity_np support for Xenomai is infeasible due to
glibc compatibility issues - again: IIRC. Gilles, please correct my weak
memory on this topic.

> 
> If not, when I "sched_setaffinity" a parent process, I have to 
> recursively "sched_setaffinity" all its threads and force them to
> switch to primary mode. Is it right ?

If you want well-defined target CPUs, then you need explicit
sched_setaffinity, cpusets, or Xenomai 2.4's global affinity mask. But
you don't need to switch back to primary mode, that will happen
automatically.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> RAKOTOSALAMA, Nirilanto wrote:
> 
>>Thanks Jan,
>>
>>It's more and more clear.
>>Just a question, I've a little doubt. Native tasks do not inherit
>>affinity from their parent process. I've read that cpu affinity 
>>setting at pthread creation is only availablle in kernel space.
>>Do Posix skin threads inherit affinity from their parent, in user space?
> 
> 
> IIRC, pthread CPU affinity is not (yet) covered by the POSIX specs, thus
> the behaviour depends on Linux here. Moreover,
> pthread_attr_setaffinity_np support for Xenomai is infeasible due to
> glibc compatibility issues - again: IIRC. Gilles, please correct my weak
> memory on this topic.

Recent versions of glibc have pthread_attr_setaffinity_np as well as
pthread_setaffinity_np. Implementing pthread_attr_setaffinity_np would
mean that we change the pthread_attr_t structure, whereas we currently
use glibc pthread_attr_t, but implementing pthread_setaffinity_np would
be feasible. However, you can use whatever service glibc provides, it
should work with Xenomai.

The only advantage of Xenomai nucleus migration service
(xnpod_migrate_thread) over Linux service (set_cpus_allowed), is that
xnpod_migrate_thread migrates the thread timers as well.

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[RAKOTOSALAMA, Nirilanto]

> > > Hi,
> > > 
> > > I'm still blocked on a CPU affinity problem.
> > > In order to adapt a set affinity function which is based on
> > > posix linux lib :
> > > - CPU_AssignPID(uint32 PID, uint32 CPU_id)
> > > - the cpu affinity of the caller and all its child threads 
> > must be set to CPU_id.
> > > 
> > > Problems are:
> > > Child PIDs must be listed, the only means I found is 
> > listing pids using `ls /proc/"Parent pid"/ > temp_file`
> > > And each listed pid is sched_setaffinity'ed.
> > > I don't know if setting affinity of RT threads from an 
> > other thread (parent) using pid works with xenomai.
> > > 
> > > So, my question is :
> > > With xenomai, is recursively cpu affinity setting from a 
> > parent thread, a good way of doing ?
> > > I read switchtest program, and I conclude that in a xenomai 
> > and RT perspective, it seems "nicer"
> > > to set affinity each threads separately during their init 
> > phase before the RT infinite loop.
> > > Otherwise, setting affinity after child threads creation 
> > from its parent, may switch them into 
> > > secondary mode during their RT loop, and at an unknown moment.
> > > Argumentation is important for my internship because I have 
> > to port on xenomai a very big RT 
> > > posix based application. And I should justify any 
> > modifications and prevent potential problems.
> > > 
> > > Sorry, I don't know if it is clear.
> > 
> > I think that it is safe to assume that if the affinity of a 
> thread is
> > inherited from the thread that created it with Linux posix 
> > library, the
> > same will happen with Xenomai.
> > 
> > Now, if you want to set the affinity of a group of threads 
> after their
> > creation, I do not see how to do this without walking 
> through the list
> > of threads. If your library has an abstraction for threads, 
> > you can keep
> > them in a list. It will be useful at process exit time to 
> > cleanly cancel
> > all threads (this may be useful if you want to detect leaks 
> > for instance).
> > 
> > Note that when talking about scheduling affinity function, we are
> > swimming in the undefined, since these functions are not 
> > standardized by
> > posix.
> > 
> > But since you are porting a library to Xenomai maybe you can 
> > have a look
> > at the current implementation of CPU_AssignPID ?
> > 
> > -- 
> >                                                  Gilles Chanteperdrix
> 
> Thanks,
> Precisly, the current implementation has nothing curious and 
> there is no reason it does not work on xenomai but,
> in a simple program that just switches the cpu :
> Before CPU_assignpid call, 'ps -ax -0 SCHED | grep <pid>'
> and '/proc/xenomai/sched' returns the same result (i.e. the 
> current thread runs on the same cpu)
> After CPU_assignpid call, 'ps -ax -0 SCHED | grep <pid>' 
> shows that threads cpu has switched
> whereas '/proc/xenomai/sched' shows no changes.
> 
> Does '/proc/xenomai/sched' not change during running, if yes
> I continue debugging my program ?

Hi,

I run with gdb a test program that only create a 1 sec periodic thread that 
rt_timer_spin's during 10ms and waits. 
As above-written, when cpu is assigned _after_ child thread creation,
'ps -ax -0 SCHED | grep <pid>' shows a cpu switch on main thread whereas,
'/proc/xenomai/sched' doesn't.
When cpu is assigned _before_ child thread creation,
'ps -ax -0 SCHED | grep <pid>' shows a cpu switch on main thread,
and '/proc/xenomai/sched' shows no changes for main thread but
the child thread cpu is the assigned one.

Is it right to supposed that, cpu switching is effective according to
'ps -ax -0 SCHED | grep <pid>' even in xenomai context ?
Does '/proc/xenomai/sched' not change during running ?
In this case, how can children threads cpu status be read dynamically like
'ps -ax -0 SCHED | grep <pid>' command for main thread ?

Thanks.
Niry


This e-mail is intended only for the above addressee. It may contain privileged information.
If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. 
If you have received it in error please delete it and immediately notify the sender.
Security Notice: all e-mail, sent to or from this address, may be accessed by someone other than the recipient, for system management and security reasons. This access is controlled under Regulation of security reasons.
This access is controlled under Regulation of Investigatory Powers Act 2000, Lawful Business Practises.

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[RAKOTOSALAMA, Nirilanto]

> > Hi,
> > 
> > I'm still blocked on a CPU affinity problem.
> > In order to adapt a set affinity function which is based on
> > posix linux lib :
> > - CPU_AssignPID(uint32 PID, uint32 CPU_id)
> > - the cpu affinity of the caller and all its child threads 
> must be set to CPU_id.
> > 
> > Problems are:
> > Child PIDs must be listed, the only means I found is 
> listing pids using `ls /proc/"Parent pid"/ > temp_file`
> > And each listed pid is sched_setaffinity'ed.
> > I don't know if setting affinity of RT threads from an 
> other thread (parent) using pid works with xenomai.
> > 
> > So, my question is :
> > With xenomai, is recursively cpu affinity setting from a 
> parent thread, a good way of doing ?
> 
> Neither good nor bad. The (i.e. your == Airbus code) library you are
> trying to adapt _wants_ to set the CPU affinity for an entire process
> hierarchy, probably to remain compatible with the behaviour inherited
> from legacy RT environments. Therefore, unless you want to change this
> behaviour at interface specification level (unlikely, 
> right?), you need
> your code to recurse down the process tree too.
>
> > I read switchtest program, and I conclude that in a xenomai 
> and RT perspective, it seems "nicer"
> > to set affinity each threads separately during their init 
> phase before the RT infinite loop.
> 
> There are two questions here:
> 
> Q. whether setting affinity during the init phase is best
> A. yes, since you want to use the regular 
> sched_setaffinity(2) call for
> that, this will switch the caller to secondary mode, and you 
> don't want
> such mode switch to happen during the actual RT processing.
> 
> Q. whether you should opt for flat or process tree-based 
> affinity setup.
> A. it depends on your requirements. IIUC, your specs imply recursive
> setup (because you want to mimic some existing behaviour), so 
> the point
> seems moot here. However, it is always better not to rely on implicit
> affinities; you may end up with a sub-optimal runtime configuration,
> just because the existing process hierarchy may not always be 
> consistent
> with the optimal CPU placement. This said, again, you might have no
> choice here if you need to port some existing library without
> significant changes, especially wrt the higher level code using such
> library.
> 
> > Otherwise, setting affinity after child threads creation 
> > from its parent, may switch them into 
> > secondary mode during their RT loop, and at an unknown moment.
> 
> Actually, only the caller (i.e. the one who calls set_schedaffinity)
> would switch to reenter the Linux kernel properly, in order to submit
> the sched_setaffinity() request. When a thread running in primary mode
> is set a different CPU affinity by another thread using the regular
> Linux kernel service (here, the former thread must be aslept 
> waiting for
> some Xenomai resource), then the new affinity would be enforced only
> when the target thread leaves primary mode. For an immediate migration
> to take place while running in primary mode, you need to ask 
> Xenomai to
> manage it, but there is no direct interface from the POSIX skin to the
> nucleus service performing this magic.
> 
> To sum up, with the Xenomai POSIX interface, you need to call
> sched_setaffinity(2) to set the CPU affinity for both Xenomai and
> regular threads, and if you want to recurse down a process hierarchy,
> you need to make sure that all the targeted children are waiting in a
> plain Linux context, for some master thread to ask for such global
> migration. For instance, you could just sync them all on some init
> barrier which would mark the end of the initialization phase, and the
> latter would include setting the CPU affinity.
> 
> This said, CPU affinity is a task property inherited upon fork, so if
> you manage to have the parent of the process tree setting up the right
> affinity before creating any child thread, you are basically done.
> 
> > Argumentation is important for my internship because I have 
> to port on xenomai a very big RT 
> > posix based application. And I should justify any 
> modifications and prevent potential problems.
> > 
> > Sorry, I don't know if it is clear.
> > 
> > Thanks in advance, Niry.
>
> Philippe.
> 

Thanks for your answer. You exactly sum up my situation (better than myself).
So, if I well undestood, the current implementation of the set affinity 
function for standard linux lib should work for xenomai threads.
So, main changes would take place into libcpu user programs.
I must take care that, in the program that uses this cpu lib, target 
children threads must "wait in linux context".
Should they "wait" sleeping (e.g. on semaphore) or sleeping avoid switching 
to secondary mode ?
Or should they wait on a simple while(!init_finished)?
Does sched_setaffinity from a master thread to a child thread block
master thread until child affinity is set ? In this instance,
sync is easy. If not ???

And if 2nd mode switching cannot be forced, this wait period 
could take a long time, could'nt it?

Niry

This e-mail is intended only for the above addressee. It may contain privileged information.
If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. 
If you have received it in error please delete it and immediately notify the sender.
Security Notice: all e-mail, sent to or from this address, may be accessed by someone other than the recipient, for system management and security reasons. This access is controlled under Regulation of security reasons.
This access is controlled under Regulation of Investigatory Powers Act 2000, Lawful Business Practises.

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[RAKOTOSALAMA, Nirilanto]

> > Hi,
> > 
> > I'm still blocked on a CPU affinity problem.
> > In order to adapt a set affinity function which is based on
> > posix linux lib :
> > - CPU_AssignPID(uint32 PID, uint32 CPU_id)
> > - the cpu affinity of the caller and all its child threads 
> must be set to CPU_id.
> > 
> > Problems are:
> > Child PIDs must be listed, the only means I found is 
> listing pids using `ls /proc/"Parent pid"/ > temp_file`
> > And each listed pid is sched_setaffinity'ed.
> > I don't know if setting affinity of RT threads from an 
> other thread (parent) using pid works with xenomai.
> > 
> > So, my question is :
> > With xenomai, is recursively cpu affinity setting from a 
> parent thread, a good way of doing ?
> > I read switchtest program, and I conclude that in a xenomai 
> and RT perspective, it seems "nicer"
> > to set affinity each threads separately during their init 
> phase before the RT infinite loop.
> > Otherwise, setting affinity after child threads creation 
> from its parent, may switch them into 
> > secondary mode during their RT loop, and at an unknown moment.
> > Argumentation is important for my internship because I have 
> to port on xenomai a very big RT 
> > posix based application. And I should justify any 
> modifications and prevent potential problems.
> > 
> > Sorry, I don't know if it is clear.
> 
> I think that it is safe to assume that if the affinity of a thread is
> inherited from the thread that created it with Linux posix 
> library, the
> same will happen with Xenomai.
> 
> Now, if you want to set the affinity of a group of threads after their
> creation, I do not see how to do this without walking through the list
> of threads. If your library has an abstraction for threads, 
> you can keep
> them in a list. It will be useful at process exit time to 
> cleanly cancel
> all threads (this may be useful if you want to detect leaks 
> for instance).
> 
> Note that when talking about scheduling affinity function, we are
> swimming in the undefined, since these functions are not 
> standardized by
> posix.
> 
> But since you are porting a library to Xenomai maybe you can 
> have a look
> at the current implementation of CPU_AssignPID ?
> 
> -- 
>                                                  Gilles Chanteperdrix

Thanks,
Precisly, the current implementation has nothing curious and 
there is no reason it does not work on xenomai but,
in a simple program that just switches the cpu :
Before CPU_assignpid call, 'ps -ax -0 SCHED | grep <pid>'
and '/proc/xenomai/sched' returns the same result (i.e. the current thread runs on the same cpu)
After CPU_assignpid call, 'ps -ax -0 SCHED | grep <pid>' shows that threads cpu has switched
whereas '/proc/xenomai/sched' shows no changes.

Does '/proc/xenomai/sched' not change during running, if yes
I continue debugging my program ?

And unfortunatly, this library does not have a threads abstraction.

CPU_Error CPU_AssignPid(uint32 Pid, CPU_CpuId Cpu)
{
  int mask = 1, i, nbTask;
  int cpu_nb;
  char cmd[SIZE_CMD] ;
  uint32 taskPid;
  FILE * filePid;
  
  cpu_nb = CPU_GetNbCpu();	
  
  if(Cpu > cpu_nb )
    {
      return CPU_INVALID_CPUID;
    }
  for (i=0; i<Cpu; i++) mask = mask *2;
  
  
  sprintf(cmd,"ls /proc/%d/task | wc | awk '{print $1}' > /tmp/nbTask.tmp",Pid);
  system(cmd);
  filePid = fopen("/tmp/nbTask.tmp","r");
  fscanf(filePid,"%d",&nbTask);
  fclose ( filePid );
  
  if ( nbTask == 1)
    {
      if(sched_setaffinity(Pid,sizeof(int), (unsigned long *)&mask) == -1)
	{/* manage error*/}
    }
  else
    {
      sprintf(cmd,"ls /proc/%d/task  > /tmp/listPid.tmp",Pid);
      system(cmd);
      filePid = fopen("/tmp/listPid.tmp","r");
      for ( i =0 ; i < nbTask ; i++ )
	{
	  fscanf(filePid,"%u",&taskPid);
	  if(sched_setaffinity(taskPid, (unsigned long *)&mask) == -1)
	   {/* manage error*/}
      fclose ( filePid );
    }  
  return CPU_OK;
}

This e-mail is intended only for the above addressee. It may contain privileged information.
If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. 
If you have received it in error please delete it and immediately notify the sender.
Security Notice: all e-mail, sent to or from this address, may be accessed by someone other than the recipient, for system management and security reasons. This access is controlled under Regulation of security reasons.
This access is controlled under Regulation of Investigatory Powers Act 2000, Lawful Business Practises.

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[RAKOTOSALAMA, Nirilanto]

Hi,

I'm still blocked on a CPU affinity problem.
In order to adapt a set affinity function which is based on
posix linux lib :
- CPU_AssignPID(uint32 PID, uint32 CPU_id)
- the cpu affinity of the caller and all its child threads must be set to CPU_id.

Problems are:
Child PIDs must be listed, the only means I found is listing pids using `ls /proc/"Parent pid"/ > temp_file`
And each listed pid is sched_setaffinity'ed.
I don't know if setting affinity of RT threads from an other thread (parent) using pid works with xenomai.

So, my question is :
With xenomai, is recursively cpu affinity setting from a parent thread, a good way of doing ?
I read switchtest program, and I conclude that in a xenomai and RT perspective, it seems "nicer"
to set affinity each threads separately during their init phase before the RT infinite loop.
Otherwise, setting affinity after child threads creation from its parent, may switch them into 
secondary mode during their RT loop, and at an unknown moment.
Argumentation is important for my internship because I have to port on xenomai a very big RT 
posix based application. And I should justify any modifications and prevent potential problems.

Sorry, I don't know if it is clear.

Thanks in advance, Niry.

This e-mail is intended only for the above addressee. It may contain privileged information.
If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. 
If you have received it in error please delete it and immediately notify the sender.
Security Notice: all e-mail, sent to or from this address, may be accessed by someone other than the recipient, for system management and security reasons. This access is controlled under Regulation of security reasons.
This access is controlled under Regulation of Investigatory Powers Act 2000, Lawful Business Practises.

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Gilles Chanteperdrix]

RAKOTOSALAMA, Nirilanto wrote:
> Hi,
> 
> I'm still blocked on a CPU affinity problem.
> In order to adapt a set affinity function which is based on
> posix linux lib :
> - CPU_AssignPID(uint32 PID, uint32 CPU_id)
> - the cpu affinity of the caller and all its child threads must be set to CPU_id.
> 
> Problems are:
> Child PIDs must be listed, the only means I found is listing pids using `ls /proc/"Parent pid"/ > temp_file`
> And each listed pid is sched_setaffinity'ed.
> I don't know if setting affinity of RT threads from an other thread (parent) using pid works with xenomai.
> 
> So, my question is :
> With xenomai, is recursively cpu affinity setting from a parent thread, a good way of doing ?
> I read switchtest program, and I conclude that in a xenomai and RT perspective, it seems "nicer"
> to set affinity each threads separately during their init phase before the RT infinite loop.
> Otherwise, setting affinity after child threads creation from its parent, may switch them into 
> secondary mode during their RT loop, and at an unknown moment.
> Argumentation is important for my internship because I have to port on xenomai a very big RT 
> posix based application. And I should justify any modifications and prevent potential problems.
> 
> Sorry, I don't know if it is clear.

I think that it is safe to assume that if the affinity of a thread is
inherited from the thread that created it with Linux posix library, the
same will happen with Xenomai.

Now, if you want to set the affinity of a group of threads after their
creation, I do not see how to do this without walking through the list
of threads. If your library has an abstraction for threads, you can keep
them in a list. It will be useful at process exit time to cleanly cancel
all threads (this may be useful if you want to detect leaks for instance).

Note that when talking about scheduling affinity function, we are
swimming in the undefined, since these functions are not standardized by
posix.

But since you are porting a library to Xenomai maybe you can have a look
at the current implementation of CPU_AssignPID ?

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2128 bytes --]

RAKOTOSALAMA, Nirilanto wrote:
> Hi,
> 
> I'm still blocked on a CPU affinity problem.
> In order to adapt a set affinity function which is based on
> posix linux lib :
> - CPU_AssignPID(uint32 PID, uint32 CPU_id)
> - the cpu affinity of the caller and all its child threads must be set to CPU_id.
> 
> Problems are:
> Child PIDs must be listed, the only means I found is listing pids using `ls /proc/"Parent pid"/ > temp_file`
> And each listed pid is sched_setaffinity'ed.
> I don't know if setting affinity of RT threads from an other thread (parent) using pid works with xenomai.

It works in so far as the Xenomai thread will not change its CPU until
it enters secondary mode. Moreover, there is no explicit "hey, RT thread
x, go to CPU #n!" under the POSIX skin. Migration always go through
secondary mode.

> 
> So, my question is :
> With xenomai, is recursively cpu affinity setting from a parent thread, a good way of doing ?
> I read switchtest program, and I conclude that in a xenomai and RT perspective, it seems "nicer"
> to set affinity each threads separately during their init phase before the RT infinite loop.
> Otherwise, setting affinity after child threads creation from its parent, may switch them into 
> secondary mode during their RT loop, and at an unknown moment.

Actually, this issue has nothing to do with Xenomai's internals. It's
rather a generic question for RT apps: Shall some thread be forced to
migrate on a different CPU while it is running? Note that this can cause
unexpected latency hickups.

I wouldn't allow such a scheme via your middleware. Either you have
clear pre-operational states of your threads in which you can change
affinity safely (also across process boundaries - if this is required),
or try to establish a different CPU assignment scheme.

> Argumentation is important for my internship because I have to port on xenomai a very big RT 
> posix based application. And I should justify any modifications and prevent potential problems.
> 
> Sorry, I don't know if it is clear.
> 
> Thanks in advance, Niry.
> 

HTH,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> RAKOTOSALAMA, Nirilanto wrote:
> 
>>Hi,
>>
>>I'm still blocked on a CPU affinity problem.
>>In order to adapt a set affinity function which is based on
>>posix linux lib :
>>- CPU_AssignPID(uint32 PID, uint32 CPU_id)
>>- the cpu affinity of the caller and all its child threads must be set to CPU_id.
>>
>>Problems are:
>>Child PIDs must be listed, the only means I found is listing pids using `ls /proc/"Parent pid"/ > temp_file`
>>And each listed pid is sched_setaffinity'ed.
>>I don't know if setting affinity of RT threads from an other thread (parent) using pid works with xenomai.
> 
> 
> It works in so far as the Xenomai thread will not change its CPU until
> it enters secondary mode. Moreover, there is no explicit "hey, RT thread
> x, go to CPU #n!" under the POSIX skin. Migration always go through
> secondary mode.

Let's repeat it, rephrased, once again: what works with Linux regarding
CPU affinity also works with Xenomai.

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> RAKOTOSALAMA, Nirilanto wrote:
>>
>>> Hi,
>>>
>>> I'm still blocked on a CPU affinity problem.
>>> In order to adapt a set affinity function which is based on
>>> posix linux lib :
>>> - CPU_AssignPID(uint32 PID, uint32 CPU_id)
>>> - the cpu affinity of the caller and all its child threads must be set to CPU_id.
>>>
>>> Problems are:
>>> Child PIDs must be listed, the only means I found is listing pids using `ls /proc/"Parent pid"/ > temp_file`
>>> And each listed pid is sched_setaffinity'ed.
>>> I don't know if setting affinity of RT threads from an other thread (parent) using pid works with xenomai.
>>
>> It works in so far as the Xenomai thread will not change its CPU until
>> it enters secondary mode. Moreover, there is no explicit "hey, RT thread
>> x, go to CPU #n!" under the POSIX skin. Migration always go through
>> secondary mode.
> 
> Let's repeat it, rephrased, once again: what works with Linux regarding
> CPU affinity also works with Xenomai.

To my understanding, there is difference in the behaviour of
sched_setaffinity() when used on a foreign pid/tid depending on if the
target is a Xenomai thread in primary mode or if it is a Linux thread,
no? If this matters in practice, that's a different question (I don't
think it does, for the reasons given earlier).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
> 
>>Jan Kiszka wrote:
>>
>>>RAKOTOSALAMA, Nirilanto wrote:
>>>
>>>
>>>>Hi,
>>>>
>>>>I'm still blocked on a CPU affinity problem.
>>>>In order to adapt a set affinity function which is based on
>>>>posix linux lib :
>>>>- CPU_AssignPID(uint32 PID, uint32 CPU_id)
>>>>- the cpu affinity of the caller and all its child threads must be set to CPU_id.
>>>>
>>>>Problems are:
>>>>Child PIDs must be listed, the only means I found is listing pids using `ls /proc/"Parent pid"/ > temp_file`
>>>>And each listed pid is sched_setaffinity'ed.
>>>>I don't know if setting affinity of RT threads from an other thread (parent) using pid works with xenomai.
>>>
>>>It works in so far as the Xenomai thread will not change its CPU until
>>>it enters secondary mode. Moreover, there is no explicit "hey, RT thread
>>>x, go to CPU #n!" under the POSIX skin. Migration always go through
>>>secondary mode.
>>
>>Let's repeat it, rephrased, once again: what works with Linux regarding
>>CPU affinity also works with Xenomai.
> 
> 
> To my understanding, there is difference in the behaviour of
> sched_setaffinity() when used on a foreign pid/tid depending on if the
> target is a Xenomai thread in primary mode or if it is a Linux thread,
> no? If this matters in practice, that's a different question (I don't
> think it does, for the reasons given earlier).

True. What we would need is an I-pipe event generated when setting the
affinity of a thread which has the PF_EVNOTIFY flag, we would handle
this event in shadow.c by calling xnpod_migrate_thread. But we need an
asynchronous event to be posted to the thread, for the case when we do
not migrate the current thread.

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Gilles Chanteperdrix]

RAKOTOSALAMA, Nirilanto wrote:
> Hi,
> 
> I'm still blocked on a CPU affinity problem.
> In order to adapt a set affinity function which is based on
> posix linux lib :
> - CPU_AssignPID(uint32 PID, uint32 CPU_id)
> - the cpu affinity of the caller and all its child threads must be set to CPU_id.
> 
> Problems are:
> Child PIDs must be listed, the only means I found is listing pids using `ls /proc/"Parent pid"/ > temp_file`
> And each listed pid is sched_setaffinity'ed.
> I don't know if setting affinity of RT threads from an other thread (parent) using pid works with xenomai.
> 
> So, my question is :
> With xenomai, is recursively cpu affinity setting from a parent thread, a good way of doing ?
> I read switchtest program, and I conclude that in a xenomai and RT perspective, it seems "nicer"
> to set affinity each threads separately during their init phase before the RT infinite loop.
> Otherwise, setting affinity after child threads creation from its parent, may switch them into 
> secondary mode during their RT loop, and at an unknown moment.
> Argumentation is important for my internship because I have to port on xenomai a very big RT 
> posix based application. And I should justify any modifications and prevent potential problems.
> 
> Sorry, I don't know if it is clear.

I think that it is safe to assume that if the affinity of a thread is
inherited from the thread that created it with Linux posix library, the
same will happen with Xenomai.

Now, if you want to set the affinity of a group of threads after their
creation, I do not see how to do this without walking through the list
of threads. If your library has an abstraction for threads, you can keep
them in a list. It will be useful at process exit time to cleanly cancel
all threads (this may be useful if you want to detect leaks for instance).

Note that when talking about scheduling affinity function, we are
swimming in the undefined, since these functions are not standardized by
posix.

But since you are porting a library to Xenomai maybe you can have a look
at the current implementation of CPU_AssignPID ?

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Philippe Gerum]

On Thu, 2007-05-31 at 15:08 +0200, RAKOTOSALAMA, Nirilanto wrote:
> Hi,
> 
> I'm still blocked on a CPU affinity problem.
> In order to adapt a set affinity function which is based on
> posix linux lib :
> - CPU_AssignPID(uint32 PID, uint32 CPU_id)
> - the cpu affinity of the caller and all its child threads must be set to CPU_id.
> 
> Problems are:
> Child PIDs must be listed, the only means I found is listing pids using `ls /proc/"Parent pid"/ > temp_file`
> And each listed pid is sched_setaffinity'ed.
> I don't know if setting affinity of RT threads from an other thread (parent) using pid works with xenomai.
> 
> So, my question is :
> With xenomai, is recursively cpu affinity setting from a parent thread, a good way of doing ?

Neither good nor bad. The (i.e. your == Airbus code) library you are
trying to adapt _wants_ to set the CPU affinity for an entire process
hierarchy, probably to remain compatible with the behaviour inherited
from legacy RT environments. Therefore, unless you want to change this
behaviour at interface specification level (unlikely, right?), you need
your code to recurse down the process tree too.

> I read switchtest program, and I conclude that in a xenomai and RT perspective, it seems "nicer"
> to set affinity each threads separately during their init phase before the RT infinite loop.

There are two questions here:

Q. whether setting affinity during the init phase is best
A. yes, since you want to use the regular sched_setaffinity(2) call for
that, this will switch the caller to secondary mode, and you don't want
such mode switch to happen during the actual RT processing.

Q. whether you should opt for flat or process tree-based affinity setup.
A. it depends on your requirements. IIUC, your specs imply recursive
setup (because you want to mimic some existing behaviour), so the point
seems moot here. However, it is always better not to rely on implicit
affinities; you may end up with a sub-optimal runtime configuration,
just because the existing process hierarchy may not always be consistent
with the optimal CPU placement. This said, again, you might have no
choice here if you need to port some existing library without
significant changes, especially wrt the higher level code using such
library.

> Otherwise, setting affinity after child threads creation from its parent, may switch them into 
> secondary mode during their RT loop, and at an unknown moment.

Actually, only the caller (i.e. the one who calls set_schedaffinity)
would switch to reenter the Linux kernel properly, in order to submit
the sched_setaffinity() request. When a thread running in primary mode
is set a different CPU affinity by another thread using the regular
Linux kernel service (here, the former thread must be aslept waiting for
some Xenomai resource), then the new affinity would be enforced only
when the target thread leaves primary mode. For an immediate migration
to take place while running in primary mode, you need to ask Xenomai to
manage it, but there is no direct interface from the POSIX skin to the
nucleus service performing this magic.

To sum up, with the Xenomai POSIX interface, you need to call
sched_setaffinity(2) to set the CPU affinity for both Xenomai and
regular threads, and if you want to recurse down a process hierarchy,
you need to make sure that all the targeted children are waiting in a
plain Linux context, for some master thread to ask for such global
migration. For instance, you could just sync them all on some init
barrier which would mark the end of the initialization phase, and the
latter would include setting the CPU affinity.

This said, CPU affinity is a task property inherited upon fork, so if
you manage to have the parent of the process tree setting up the right
affinity before creating any child thread, you are basically done.

> Argumentation is important for my internship because I have to port on xenomai a very big RT 
> posix based application. And I should justify any modifications and prevent potential problems.
> 
> Sorry, I don't know if it is clear.
> 
> Thanks in advance, Niry.
> 
> This e-mail is intended only for the above addressee. It may contain privileged information.
> If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. 
> If you have received it in error please delete it and immediately notify the sender.
> Security Notice: all e-mail, sent to or from this address, may be accessed by someone other than the recipient, for system management and security reasons. This access is controlled under Regulation of security reasons.
> This access is controlled under Regulation of Investigatory Powers Act 2000, Lawful Business Practises.
> 
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[RAKOTOSALAMA, Nirilanto]

> > Hello,
> > 
> > I have some question about CPU affinity with Xenomai 2.3.1.
> > (it is to see if  xenomai support developing for a cpu 
> manager library is feasible)
> 
> What will this library do? (/me just nosy)

In fact, there's simulation programs that run on many different systems (hard + soft),
and to make this applications as portable as possible, they use libraries developed for every
systems (dec alpha, pc, specific cards, linux, unix). And, I search if xenomai could be used
with less re-implementation effort, so the more posix function are developed by xenomai, 
the better it is, and easier it is (for me :p).
So first step is developing this libraries xenomai support (e.g. timer, cpu).
This cpu library should provides "simple" functions like : getting the number of cpu,
setting/getting the cpu affinity for a process passing its pid, protect/un-protect cpu from running
unwanted process.

> > When using rt_task_create, CPU mask can be specified. 
> However, when sched_setaffinity is called 
> > to modify main program affinity, and after, rt_task_create 
> is called without specifying CPU mask,
> > did the child task inherits from the main program affinity ?
> 
> If you don't provide a target CPU (either on creation or via
> /proc/xenomai/affinity), better consider the affinity as unspecified
> (any possible CPU). The target CPU derives from Linux 
> decision where to
> put the thread non-RT boot code on, Xenomai just nails that decision
> down until someone explicitly migrates.
> 
> > 
> > Same question for Posix skin threads ?
> > 
> > Can native task affinity be set other than when creating ? 
> using sdhed_setaffinity ?
> 
> /proc/xenomai/affinity (has global scope) or 
> sched_setaffinity. Xenomai
> shadow threads follow the migration that may took place while in
> secondary mode, and sched_setaffinity comes with such a mode switch.

Thanks, If I have well understood, tasks or threads affinity can be set using /proc/xenomai/affinity
but, I have read that /proc/xenomai/affinity does not exist in xenomai 2.3.1 and xenomai must be patched
(I don't know how because I don't have nor SVN neither internet connection on my xenomai boxes).
So I can use sched_setaffinity, but it make switch to secondary mode. 
Does'nt matter if only threads must be RT and sched_setaffinity is called during a non-rt init phase, does it ?
That means that affinity does'nt depend on running on xenomai or secondary mode ?

BTW, is protecting a CPU with xenomai possible ? 
Does xenomai use /dev/cpuset or an other device ?

> > Can posix skin thread affinity be set using sched_affinity ?
> > 
> > 
> > Thanks in advance.
> > 
> > Niry.
> > 
> 
> Jan
> 
Niry

This e-mail is intended only for the above addressee. It may contain privileged information.
If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. 
If you have received it in error please delete it and immediately notify the sender.
Security Notice: all e-mail, sent to or from this address, may be accessed by someone other than the recipient, for system management and security reasons. This access is controlled under Regulation of security reasons.
This access is controlled under Regulation of Investigatory Powers Act 2000, Lawful Business Practises.

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3083 bytes --]

RAKOTOSALAMA, Nirilanto wrote:
>>> Hello,
>>>
>>> I have some question about CPU affinity with Xenomai 2.3.1.
>>> (it is to see if  xenomai support developing for a cpu 
>> manager library is feasible)
>>
>> What will this library do? (/me just nosy)
> 
> In fact, there's simulation programs that run on many different systems (hard + soft),
> and to make this applications as portable as possible, they use libraries developed for every
> systems (dec alpha, pc, specific cards, linux, unix). And, I search if xenomai could be used
> with less re-implementation effort, so the more posix function are developed by xenomai, 
> the better it is, and easier it is (for me :p).
> So first step is developing this libraries xenomai support (e.g. timer, cpu).
> This cpu library should provides "simple" functions like : getting the number of cpu,
> setting/getting the cpu affinity for a process passing its pid, protect/un-protect cpu from running
> unwanted process.

OK, I see.

> 
>>> When using rt_task_create, CPU mask can be specified. 
>> However, when sched_setaffinity is called 
>>> to modify main program affinity, and after, rt_task_create 
>> is called without specifying CPU mask,
>>> did the child task inherits from the main program affinity ?
>> If you don't provide a target CPU (either on creation or via
>> /proc/xenomai/affinity), better consider the affinity as unspecified
>> (any possible CPU). The target CPU derives from Linux 
>> decision where to
>> put the thread non-RT boot code on, Xenomai just nails that decision
>> down until someone explicitly migrates.
>>
>>> Same question for Posix skin threads ?
>>>
>>> Can native task affinity be set other than when creating ? 
>> using sdhed_setaffinity ?
>>
>> /proc/xenomai/affinity (has global scope) or 
>> sched_setaffinity. Xenomai
>> shadow threads follow the migration that may took place while in
>> secondary mode, and sched_setaffinity comes with such a mode switch.
> 
> Thanks, If I have well understood, tasks or threads affinity can be set using /proc/xenomai/affinity
> but, I have read that /proc/xenomai/affinity does not exist in xenomai 2.3.1 and xenomai must be patched

Yes, it's an upcoming feature of 2.4.

> (I don't know how because I don't have nor SVN neither internet connection on my xenomai boxes).

There is a daily snapshot on gna.org: https://gna.org/svn/?group=xenomai

> So I can use sched_setaffinity, but it make switch to secondary mode. 
> Does'nt matter if only threads must be RT and sched_setaffinity is called during a non-rt init phase, does it ?

Yep.

> That means that affinity does'nt depend on running on xenomai or secondary mode ?

Sorry, doesn't parse for me right now.

> 
> BTW, is protecting a CPU with xenomai possible ? 
> Does xenomai use /dev/cpuset or an other device ?

Xenomai doesn't directly interact with cpuset -- but... if you force
some RT process into a certain CPU and you don't override this decision
on RT thread creation, Xenomai will follow as explained.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-help] [Newbie question] threads and task CPU affinity

[RAKOTOSALAMA, Nirilanto]

Hello,

I have some question about CPU affinity with Xenomai 2.3.1.
(it is to see if  xenomai support developing for a cpu manager library is feasible)

When using rt_task_create, CPU mask can be specified. However, when sched_setaffinity is called 
to modify main program affinity, and after, rt_task_create is called without specifying CPU mask,
did the child task inherits from the main program affinity ?

Same question for Posix skin threads ?

Can native task affinity be set other than when creating ? using sdhed_setaffinity ?

Can posix skin thread affinity be set using sched_affinity ?


Thanks in advance.

Niry.

This e-mail is intended only for the above addressee. It may contain privileged information.
If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. 
If you have received it in error please delete it and immediately notify the sender.
Security Notice: all e-mail, sent to or from this address, may be accessed by someone other than the recipient, for system management and security reasons. This access is controlled under Regulation of security reasons.
This access is controlled under Regulation of Investigatory Powers Act 2000, Lawful Business Practises.

Re: [Xenomai-help] [Newbie question] threads and task CPU affinity

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1308 bytes --]

RAKOTOSALAMA, Nirilanto wrote:
> Hello,
> 
> I have some question about CPU affinity with Xenomai 2.3.1.
> (it is to see if  xenomai support developing for a cpu manager library is feasible)

What will this library do? (/me just nosy)

> 
> When using rt_task_create, CPU mask can be specified. However, when sched_setaffinity is called 
> to modify main program affinity, and after, rt_task_create is called without specifying CPU mask,
> did the child task inherits from the main program affinity ?

If you don't provide a target CPU (either on creation or via
/proc/xenomai/affinity), better consider the affinity as unspecified
(any possible CPU). The target CPU derives from Linux decision where to
put the thread non-RT boot code on, Xenomai just nails that decision
down until someone explicitly migrates.

> 
> Same question for Posix skin threads ?
> 
> Can native task affinity be set other than when creating ? using sdhed_setaffinity ?

/proc/xenomai/affinity (has global scope) or sched_setaffinity. Xenomai
shadow threads follow the migration that may took place while in
secondary mode, and sched_setaffinity comes with such a mode switch.

> 
> Can posix skin thread affinity be set using sched_affinity ?
> 
> 
> Thanks in advance.
> 
> Niry.
> 

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]