[Xenomai] RT_TASK affinity on more than 8 CPU's

[Xenomai] RT_TASK affinity on more than 8 CPU's

[Wolz, Troy]

Hello,

I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.

Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used. Tracing through the use of this mask, it appears that the mode mask passed into rt_task_create uses the top 8 bits of the word as the cpu mask. The bits following the top 8 are used for other flags, so there is no room to expand the cpu mask.

I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.

2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.

We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.

Troy

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/16/2012 10:16 PM, Wolz, Troy wrote:
> Hello,
> 
> I'm working on a project where I'd like to start more than 8 RT_TASKs
> each affined to their own CPU. From looking at the documentation,
> using the native API it appears to be possible to set affinity to a
> CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255,
> but when setting affinity above 7, overflow occurs and only the lower
> 8 CPU's are used.
> 
> Looking into this, the T_CPU macro appears to only use the lower 3
> bits of the value passed to it, implying that T_CPU only allows up to
> 8 CPU's to be used. Tracing through the use of this mask, it appears
> that the mode mask passed into rt_task_create uses the top 8 bits of
> the word as the cpu mask. The bits following the top 8 are used for
> other flags, so there is no room to expand the cpu mask.
> 
> I see 2 solutions to this limitation for using Xenomai and affinity
> on more than 8 cpus. 1. Expand the 'mode' argument from a 32 bit int
> to a 64 bit long. With the extra bits, the top 32 bits could be used
> as a CPU mask, allowing for up to 32 cpu's to be masked
> simultaneously. This change would affect several files, but it should
> be backwards compatible with applications currently written using a
> 32 bit mode.
> 
> 2. Pass an additional 64 bit long argument to rt_task_create that
> only contains the affinity mask. In this case, the mode argument
> would be used the same as previously, except the affinity would be
> ignored. This option has the disadvantage of being incompatible with
> existing Xenomai applications, but it is very easy to set affinity
> for up to 64 cpu's. We could add a preprocessor define that selects
> whether the existing rt_task_create method is available or whether
> the new method is available.

Solution 3: use glibc API to change affinity:
sched_setaffinity or pthread_setaffinity.

Also note that Xenomai design being based on a "big kernel lock", it
will scale poorly with too many cpus.

-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Philippe Gerum]

On 07/16/2012 10:16 PM, Wolz, Troy wrote:
> Hello,
>
> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>
> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
  Tracing through the use of this mask, it appears that the mode mask 
passed into rt_task_create uses the top 8 bits of the word as the cpu 
mask. The bits following the top 8 are used for other flags, so there is 
no room to expand the cpu mask.
>
> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>
> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>
> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>

The reason not to provide for more than 8 CPUs stems from our locking 
model, based on a single big lock to protect the system data structures. 
This is a design decision to keep the real-time kernel simple and 
robust, at the expense of scalability limited to few CPUs.
Our embedded bias, and the fact that CPU proliferation is not the norm 
there yet, particularly for dual kernel configurations, explains this.

Note that this setting relates to CPUs used in real-time mode, not 
necessarily to the overall number of CPUs available. E.g. you may pin 
your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs 
system. Granted, the API currently allows real-time stuff to be pinned 
on CPUs #0-7 only. But you may change the affinity after task creation, 
using regular POSIX calls, this change will be propagated to the Xenomai 
scheduler next time the task moves back to real-time mode.
That would be the solution for Xenomai 2.x, which is exclusively based 
on a dual kernel architecture.

For the upcoming 3.x, the situation may be different, since we will have 
the same API also available on native real-time systems (e.g. 
preempt-rt). In that case, we do have a shortcoming in the way 
rt_task_create() accepts the CPU affinity setting, since there is no 
reason to restrict the range of available CPUs there.

In that case, a sane option could be to move the CPU affinity data out 
of the task mode from rt_task_create(), and provide a 
rt_task_setaffinity() call of some sort instead, receiving a cpuset map. 
This would keep the API compatible for most existing use cases, only 
requiring people who actually pin tasks to CPUs to update their code. 
However, this change won't make it in Xenomai 2.6, the API and ABI are 
frozen for this branch, which has entered the maintenance cycle.

PS: could you tune your MUA to split long lines? TIA,

> Troy
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai
>


-- 
Philippe.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/21/2012 08:13 PM, Philippe Gerum wrote:
> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>> Hello,
>>
>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>
>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>   Tracing through the use of this mask, it appears that the mode mask 
> passed into rt_task_create uses the top 8 bits of the word as the cpu 
> mask. The bits following the top 8 are used for other flags, so there is 
> no room to expand the cpu mask.
>>
>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>
>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>
>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>
> 
> The reason not to provide for more than 8 CPUs stems from our locking 
> model, based on a single big lock to protect the system data structures. 
> This is a design decision to keep the real-time kernel simple and 
> robust, at the expense of scalability limited to few CPUs.
> Our embedded bias, and the fact that CPU proliferation is not the norm 
> there yet, particularly for dual kernel configurations, explains this.
> 
> Note that this setting relates to CPUs used in real-time mode, not 
> necessarily to the overall number of CPUs available. E.g. you may pin 
> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs 
> system.

I am afraid for the system to scale poorly, only timer interrupt are
needed: the timer interrupt handler takes the big lock whether it does
anything else than handling the linux timer or not. So, if timer
interrupts happen at the same time on all cpus, we are going to observe
big latencies, without the help of real-time tasks running on more than
one cpu.

-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Philippe Gerum]

On 07/21/2012 09:21 PM, Gilles Chanteperdrix wrote:
> On 07/21/2012 08:13 PM, Philippe Gerum wrote:
>> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>>> Hello,
>>>
>>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>>
>>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>>    Tracing through the use of this mask, it appears that the mode mask
>> passed into rt_task_create uses the top 8 bits of the word as the cpu
>> mask. The bits following the top 8 are used for other flags, so there is
>> no room to expand the cpu mask.
>>>
>>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>>
>>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>>
>>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>>
>>
>> The reason not to provide for more than 8 CPUs stems from our locking
>> model, based on a single big lock to protect the system data structures.
>> This is a design decision to keep the real-time kernel simple and
>> robust, at the expense of scalability limited to few CPUs.
>> Our embedded bias, and the fact that CPU proliferation is not the norm
>> there yet, particularly for dual kernel configurations, explains this.
>>
>> Note that this setting relates to CPUs used in real-time mode, not
>> necessarily to the overall number of CPUs available. E.g. you may pin
>> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs
>> system.
>
> I am afraid for the system to scale poorly, only timer interrupt are
> needed: the timer interrupt handler takes the big lock whether it does
> anything else than handling the linux timer or not. So, if timer
> interrupts happen at the same time on all cpus, we are going to observe
> big latencies, without the help of real-time tasks running on more than
> one cpu.
>

You are assuming that the Xenomai clock event device must be common to 
the linux one, based on our host timer relay mechanism, I'm not. For 
these high end configurations, we'd have to pick a dedicated timer 
hardware for Xenomai.

-- 
Philippe.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/22/2012 04:15 PM, Philippe Gerum wrote:
> On 07/21/2012 09:21 PM, Gilles Chanteperdrix wrote:
>> On 07/21/2012 08:13 PM, Philippe Gerum wrote:
>>> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>>>> Hello,
>>>>
>>>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>>>
>>>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>>>    Tracing through the use of this mask, it appears that the mode mask
>>> passed into rt_task_create uses the top 8 bits of the word as the cpu
>>> mask. The bits following the top 8 are used for other flags, so there is
>>> no room to expand the cpu mask.
>>>>
>>>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>>>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>>>
>>>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>>>
>>>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>>>
>>>
>>> The reason not to provide for more than 8 CPUs stems from our locking
>>> model, based on a single big lock to protect the system data structures.
>>> This is a design decision to keep the real-time kernel simple and
>>> robust, at the expense of scalability limited to few CPUs.
>>> Our embedded bias, and the fact that CPU proliferation is not the norm
>>> there yet, particularly for dual kernel configurations, explains this.
>>>
>>> Note that this setting relates to CPUs used in real-time mode, not
>>> necessarily to the overall number of CPUs available. E.g. you may pin
>>> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs
>>> system.
>>
>> I am afraid for the system to scale poorly, only timer interrupt are
>> needed: the timer interrupt handler takes the big lock whether it does
>> anything else than handling the linux timer or not. So, if timer
>> interrupts happen at the same time on all cpus, we are going to observe
>> big latencies, without the help of real-time tasks running on more than
>> one cpu.
>>
> 
> You are assuming that the Xenomai clock event device must be common to 
> the linux one, based on our host timer relay mechanism, I'm not. For 
> these high end configurations, we'd have to pick a dedicated timer 
> hardware for Xenomai.

It does not work for the current implementations of the I-pipe core and
Xenomai:
- the I-pipe core assumes that it uses the same timer on all CPUS, or at
least timers with the same frequency, changing this means changing
Xenomai code.
- we have no way of saying that we want to grab timers only for some
CPUS, we grab them from 0 to num_online_cpus()
- we use the timer with the highest Linux rating, which is not in the
CLOCK_EVT_MODE_SHUTDOWN state, so we end up using the same timer as Linux.

The only way I could see this working on x86 is if Linux is using HPET
as clock event device on some CPUS, and APIC on others, and Xenomai APIC
on all cpus. But since APIC and HPET have different frequencies, we need
to change xenomai code.


-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/22/2012 04:29 PM, Gilles Chanteperdrix wrote:
> On 07/22/2012 04:15 PM, Philippe Gerum wrote:
>> On 07/21/2012 09:21 PM, Gilles Chanteperdrix wrote:
>>> On 07/21/2012 08:13 PM, Philippe Gerum wrote:
>>>> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>>>>> Hello,
>>>>>
>>>>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>>>>
>>>>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>>>>    Tracing through the use of this mask, it appears that the mode mask
>>>> passed into rt_task_create uses the top 8 bits of the word as the cpu
>>>> mask. The bits following the top 8 are used for other flags, so there is
>>>> no room to expand the cpu mask.
>>>>>
>>>>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>>>>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>>>>
>>>>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>>>>
>>>>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>>>>
>>>>
>>>> The reason not to provide for more than 8 CPUs stems from our locking
>>>> model, based on a single big lock to protect the system data structures.
>>>> This is a design decision to keep the real-time kernel simple and
>>>> robust, at the expense of scalability limited to few CPUs.
>>>> Our embedded bias, and the fact that CPU proliferation is not the norm
>>>> there yet, particularly for dual kernel configurations, explains this.
>>>>
>>>> Note that this setting relates to CPUs used in real-time mode, not
>>>> necessarily to the overall number of CPUs available. E.g. you may pin
>>>> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs
>>>> system.
>>>
>>> I am afraid for the system to scale poorly, only timer interrupt are
>>> needed: the timer interrupt handler takes the big lock whether it does
>>> anything else than handling the linux timer or not. So, if timer
>>> interrupts happen at the same time on all cpus, we are going to observe
>>> big latencies, without the help of real-time tasks running on more than
>>> one cpu.
>>>
>>
>> You are assuming that the Xenomai clock event device must be common to 
>> the linux one, based on our host timer relay mechanism, I'm not. For 
>> these high end configurations, we'd have to pick a dedicated timer 
>> hardware for Xenomai.
> 
> It does not work for the current implementations of the I-pipe core and
> Xenomai:
> - the I-pipe core assumes that it uses the same timer on all CPUS, or at
> least timers with the same frequency, changing this means changing
> Xenomai code.
> - we have no way of saying that we want to grab timers only for some
> CPUS, we grab them from 0 to num_online_cpus()
> - we use the timer with the highest Linux rating, which is not in the
> CLOCK_EVT_MODE_SHUTDOWN state, so we end up using the same timer as Linux.
> 
> The only way I could see this working on x86 is if Linux is using HPET
> as clock event device on some CPUS, and APIC on others, and Xenomai APIC
> on all cpus. But since APIC and HPET have different frequencies, we need
> to change xenomai code.

Ok. Strike that, yes, it already works.

-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Jan Kiszka]

On 2012-07-21 21:21, Gilles Chanteperdrix wrote:
> On 07/21/2012 08:13 PM, Philippe Gerum wrote:
>> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>>> Hello,
>>>
>>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>>
>>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>>   Tracing through the use of this mask, it appears that the mode mask 
>> passed into rt_task_create uses the top 8 bits of the word as the cpu 
>> mask. The bits following the top 8 are used for other flags, so there is 
>> no room to expand the cpu mask.
>>>
>>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>>
>>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>>
>>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>>
>>
>> The reason not to provide for more than 8 CPUs stems from our locking 
>> model, based on a single big lock to protect the system data structures. 
>> This is a design decision to keep the real-time kernel simple and 
>> robust, at the expense of scalability limited to few CPUs.
>> Our embedded bias, and the fact that CPU proliferation is not the norm 
>> there yet, particularly for dual kernel configurations, explains this.
>>
>> Note that this setting relates to CPUs used in real-time mode, not 
>> necessarily to the overall number of CPUs available. E.g. you may pin 
>> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs 
>> system.
> 
> I am afraid for the system to scale poorly, only timer interrupt are
> needed: the timer interrupt handler takes the big lock whether it does
> anything else than handling the linux timer or not. So, if timer
> interrupts happen at the same time on all cpus, we are going to observe
> big latencies, without the help of real-time tasks running on more than
> one cpu.

We have xeno_hal.supported_cpus to address this issue.

We are running Xenomai on boxes >= 8 cores for quite a while. RT load is
mostly confined to a single core then.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/23/2012 09:47 AM, Jan Kiszka wrote:

> On 2012-07-21 21:21, Gilles Chanteperdrix wrote:
>> On 07/21/2012 08:13 PM, Philippe Gerum wrote:
>>> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>>>> Hello,
>>>>
>>>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>>>
>>>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>>>   Tracing through the use of this mask, it appears that the mode mask 
>>> passed into rt_task_create uses the top 8 bits of the word as the cpu 
>>> mask. The bits following the top 8 are used for other flags, so there is 
>>> no room to expand the cpu mask.
>>>>
>>>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>>>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>>>
>>>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>>>
>>>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>>>
>>>
>>> The reason not to provide for more than 8 CPUs stems from our locking 
>>> model, based on a single big lock to protect the system data structures. 
>>> This is a design decision to keep the real-time kernel simple and 
>>> robust, at the expense of scalability limited to few CPUs.
>>> Our embedded bias, and the fact that CPU proliferation is not the norm 
>>> there yet, particularly for dual kernel configurations, explains this.
>>>
>>> Note that this setting relates to CPUs used in real-time mode, not 
>>> necessarily to the overall number of CPUs available. E.g. you may pin 
>>> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs 
>>> system.
>>
>> I am afraid for the system to scale poorly, only timer interrupt are
>> needed: the timer interrupt handler takes the big lock whether it does
>> anything else than handling the linux timer or not. So, if timer
>> interrupts happen at the same time on all cpus, we are going to observe
>> big latencies, without the help of real-time tasks running on more than
>> one cpu.
> 
> We have xeno_hal.supported_cpus to address this issue.
> 
> We are running Xenomai on boxes >= 8 cores for quite a while. RT load is
> mostly confined to a single core then.


Ah, missed that. But the I-pipe core should be changed then, because
ipipe_timers_request looks for and grabs timers on all cpus, switching
them to one-shot mode.

-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Philippe Gerum]

On 07/23/2012 10:08 AM, Gilles Chanteperdrix wrote:
> On 07/23/2012 09:47 AM, Jan Kiszka wrote:
>
>> On 2012-07-21 21:21, Gilles Chanteperdrix wrote:
>>> On 07/21/2012 08:13 PM, Philippe Gerum wrote:
>>>> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>>>>> Hello,
>>>>>
>>>>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>>>>
>>>>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>>>>    Tracing through the use of this mask, it appears that the mode mask
>>>> passed into rt_task_create uses the top 8 bits of the word as the cpu
>>>> mask. The bits following the top 8 are used for other flags, so there is
>>>> no room to expand the cpu mask.
>>>>>
>>>>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>>>>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>>>>
>>>>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>>>>
>>>>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>>>>
>>>>
>>>> The reason not to provide for more than 8 CPUs stems from our locking
>>>> model, based on a single big lock to protect the system data structures.
>>>> This is a design decision to keep the real-time kernel simple and
>>>> robust, at the expense of scalability limited to few CPUs.
>>>> Our embedded bias, and the fact that CPU proliferation is not the norm
>>>> there yet, particularly for dual kernel configurations, explains this.
>>>>
>>>> Note that this setting relates to CPUs used in real-time mode, not
>>>> necessarily to the overall number of CPUs available. E.g. you may pin
>>>> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs
>>>> system.
>>>
>>> I am afraid for the system to scale poorly, only timer interrupt are
>>> needed: the timer interrupt handler takes the big lock whether it does
>>> anything else than handling the linux timer or not. So, if timer
>>> interrupts happen at the same time on all cpus, we are going to observe
>>> big latencies, without the help of real-time tasks running on more than
>>> one cpu.
>>
>> We have xeno_hal.supported_cpus to address this issue.
>>
>> We are running Xenomai on boxes >= 8 cores for quite a while. RT load is
>> mostly confined to a single core then.
>
>
> Ah, missed that. But the I-pipe core should be changed then, because
> ipipe_timers_request looks for and grabs timers on all cpus, switching
> them to one-shot mode.
>

It's really an implementation detail, we could even make it a weakly 
linked routine to allow for interposing on it from the arch-dependent 
pipeline code. The bottom line is that we don't have to interpose fully 
on the linux timer machinery, we only do this because it is a simple 
default method.

So, yes, one may have to fix up the pipeline core on a per-platform 
basis, for using a different set of clock event devices specifically for 
real-time use. Some platforms provide a set of unused GP timers, or via 
a PCI watchdog+timer board extension. But this is quite regular stuff to 
provide, far from a core Xenomai change. The logic of the Xenomai timer 
sub-system already allows that. But since this can only be a 
platform-specific solution depending on a particular set of available 
timers, nobody argues this could be a generic, off-the-shelf option.

-- 
Philippe.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/23/2012 10:40 AM, Philippe Gerum wrote:

> On 07/23/2012 10:08 AM, Gilles Chanteperdrix wrote:
>> On 07/23/2012 09:47 AM, Jan Kiszka wrote:
>>
>>> On 2012-07-21 21:21, Gilles Chanteperdrix wrote:
>>>> On 07/21/2012 08:13 PM, Philippe Gerum wrote:
>>>>> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>>>>>
>>>>>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>>>>>    Tracing through the use of this mask, it appears that the mode mask
>>>>> passed into rt_task_create uses the top 8 bits of the word as the cpu
>>>>> mask. The bits following the top 8 are used for other flags, so there is
>>>>> no room to expand the cpu mask.
>>>>>>
>>>>>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>>>>>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>>>>>
>>>>>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>>>>>
>>>>>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>>>>>
>>>>>
>>>>> The reason not to provide for more than 8 CPUs stems from our locking
>>>>> model, based on a single big lock to protect the system data structures.
>>>>> This is a design decision to keep the real-time kernel simple and
>>>>> robust, at the expense of scalability limited to few CPUs.
>>>>> Our embedded bias, and the fact that CPU proliferation is not the norm
>>>>> there yet, particularly for dual kernel configurations, explains this.
>>>>>
>>>>> Note that this setting relates to CPUs used in real-time mode, not
>>>>> necessarily to the overall number of CPUs available. E.g. you may pin
>>>>> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs
>>>>> system.
>>>>
>>>> I am afraid for the system to scale poorly, only timer interrupt are
>>>> needed: the timer interrupt handler takes the big lock whether it does
>>>> anything else than handling the linux timer or not. So, if timer
>>>> interrupts happen at the same time on all cpus, we are going to observe
>>>> big latencies, without the help of real-time tasks running on more than
>>>> one cpu.
>>>
>>> We have xeno_hal.supported_cpus to address this issue.
>>>
>>> We are running Xenomai on boxes >= 8 cores for quite a while. RT load is
>>> mostly confined to a single core then.
>>
>>
>> Ah, missed that. But the I-pipe core should be changed then, because
>> ipipe_timers_request looks for and grabs timers on all cpus, switching
>> them to one-shot mode.
>>
> 
> It's really an implementation detail, we could even make it a weakly 
> linked routine to allow for interposing on it from the arch-dependent 
> pipeline code. The bottom line is that we don't have to interpose fully 
> on the linux timer machinery, we only do this because it is a simple 
> default method.
> 
> So, yes, one may have to fix up the pipeline core on a per-platform 
> basis, for using a different set of clock event devices specifically for 
> real-time use. Some platforms provide a set of unused GP timers, or via 
> a PCI watchdog+timer board extension. But this is quite regular stuff to 
> provide, far from a core Xenomai change. The logic of the Xenomai timer 
> sub-system already allows that. But since this can only be a 
> platform-specific solution depending on a particular set of available 
> timers, nobody argues this could be a generic, off-the-shelf option.
> 


As I said, the core does not allow using timers with different
frequencies on different cpus, so basically, it wants the same timer on
all cpus, this is the reason why ipipe_request_timers tries to check
that the timers which will be used on all cpus use the same frequency.

If we want to change that, we have to fix Xenomai, which supposes there
is only one "timer_freq" (rthal_tunables.timer_freq). We can probably
get away with setting RTHAL_TIMER_FREQ to a per_cpu variable on all
architectures but arm which uses the frequency to pre-compute a
multiplicand to avoid division when converting between cpu frequency and
timer frequency in xnarch_program_timer_shot.

Otherwise, we can pass a cpumask to ipipe_request_timers to tell on what
cpus we need timer.

If we do not fix that one way or another, anyone wanting to use another
timer than the default ones will have to provide a timer for each cpu core.

-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Philippe Gerum]

On 07/23/2012 10:57 AM, Gilles Chanteperdrix wrote:
> On 07/23/2012 10:40 AM, Philippe Gerum wrote:
>
>> On 07/23/2012 10:08 AM, Gilles Chanteperdrix wrote:
>>> On 07/23/2012 09:47 AM, Jan Kiszka wrote:
>>>
>>>> On 2012-07-21 21:21, Gilles Chanteperdrix wrote:
>>>>> On 07/21/2012 08:13 PM, Philippe Gerum wrote:
>>>>>> On 07/16/2012 10:16 PM, Wolz, Troy wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm working on a project where I'd like to start more than 8 RT_TASKs each affined to their own CPU. From looking at the documentation, using the native API it appears to be possible to set affinity to a CPU from 0 to RTHAL_NR_CPUS-1. On my machine, RTHAL_NR_CPUS is 255, but when setting affinity above 7, overflow occurs and only the lower 8 CPU's are used.
>>>>>>>
>>>>>>> Looking into this, the T_CPU macro appears to only use the lower 3 bits of the value passed to it, implying that T_CPU only allows up to 8 CPU's to be used.
>>>>>>     Tracing through the use of this mask, it appears that the mode mask
>>>>>> passed into rt_task_create uses the top 8 bits of the word as the cpu
>>>>>> mask. The bits following the top 8 are used for other flags, so there is
>>>>>> no room to expand the cpu mask.
>>>>>>>
>>>>>>> I see 2 solutions to this limitation for using Xenomai and affinity on more than 8 cpus.
>>>>>>> 1. Expand the 'mode' argument from a 32 bit int to a 64 bit long. With the extra bits, the top 32 bits could be used as a CPU mask, allowing for up to 32 cpu's to be masked simultaneously. This change would affect several files, but it should be backwards compatible with applications currently written using a 32 bit mode.
>>>>>>>
>>>>>>> 2. Pass an additional 64 bit long argument to rt_task_create that only contains the affinity mask. In this case, the mode argument would be used the same as previously, except the affinity would be ignored. This option has the disadvantage of being incompatible with existing Xenomai applications, but it is very easy to set affinity for up to 64 cpu's. We could add a preprocessor define that selects whether the existing rt_task_create method is available or whether the new method is available.
>>>>>>>
>>>>>>> We are willing to develop a patch that adds this functionality, but I wanted to talk it over with the group to determine if this would be accepted as a patch or if there's a better way to do it. Thanks.
>>>>>>>
>>>>>>
>>>>>> The reason not to provide for more than 8 CPUs stems from our locking
>>>>>> model, based on a single big lock to protect the system data structures.
>>>>>> This is a design decision to keep the real-time kernel simple and
>>>>>> robust, at the expense of scalability limited to few CPUs.
>>>>>> Our embedded bias, and the fact that CPU proliferation is not the norm
>>>>>> there yet, particularly for dual kernel configurations, explains this.
>>>>>>
>>>>>> Note that this setting relates to CPUs used in real-time mode, not
>>>>>> necessarily to the overall number of CPUs available. E.g. you may pin
>>>>>> your tasks on 4 CPUs dedicated to real-time processing, over a 64 CPUs
>>>>>> system.
>>>>>
>>>>> I am afraid for the system to scale poorly, only timer interrupt are
>>>>> needed: the timer interrupt handler takes the big lock whether it does
>>>>> anything else than handling the linux timer or not. So, if timer
>>>>> interrupts happen at the same time on all cpus, we are going to observe
>>>>> big latencies, without the help of real-time tasks running on more than
>>>>> one cpu.
>>>>
>>>> We have xeno_hal.supported_cpus to address this issue.
>>>>
>>>> We are running Xenomai on boxes >= 8 cores for quite a while. RT load is
>>>> mostly confined to a single core then.
>>>
>>>
>>> Ah, missed that. But the I-pipe core should be changed then, because
>>> ipipe_timers_request looks for and grabs timers on all cpus, switching
>>> them to one-shot mode.
>>>
>>
>> It's really an implementation detail, we could even make it a weakly
>> linked routine to allow for interposing on it from the arch-dependent
>> pipeline code. The bottom line is that we don't have to interpose fully
>> on the linux timer machinery, we only do this because it is a simple
>> default method.
>>
>> So, yes, one may have to fix up the pipeline core on a per-platform
>> basis, for using a different set of clock event devices specifically for
>> real-time use. Some platforms provide a set of unused GP timers, or via
>> a PCI watchdog+timer board extension. But this is quite regular stuff to
>> provide, far from a core Xenomai change. The logic of the Xenomai timer
>> sub-system already allows that. But since this can only be a
>> platform-specific solution depending on a particular set of available
>> timers, nobody argues this could be a generic, off-the-shelf option.
>>
>
>
> As I said, the core does not allow using timers with different
> frequencies on different cpus, so basically, it wants the same timer on
> all cpus, this is the reason why ipipe_request_timers tries to check
> that the timers which will be used on all cpus use the same frequency.
>

You seem to be focused on the x86 case, where diverting the per-CPU APIC 
timer is by design a PITA. I'm considering the more general case, where 
we can have a dedicated hardware already, e.g. GPTs, basically answering 
the question: does many CPUs make the system unusable for real-time by 
design? My assessment is that it does not. With common clock event 
devices not initially shared with linux, assuming a common frequency is 
not even an issue, albeit this can be considered as a useless 
restriction of the Xenomai core.

> If we want to change that, we have to fix Xenomai, which supposes there
> is only one "timer_freq" (rthal_tunables.timer_freq). We can probably
> get away with setting RTHAL_TIMER_FREQ to a per_cpu variable on all
> architectures but arm which uses the frequency to pre-compute a
> multiplicand to avoid division when converting between cpu frequency and
> timer frequency in xnarch_program_timer_shot.
>

Maybe it's time to move the hrtimer-freq <-> hrclock-freq computations 
to some globally visible helpers into the pipeline core. Then, the 
arch-dep sections of the pipeline could go wild doing these computations 
the way they want to, without affecting the interface anymore. 
Typically, there is nothing which would prevent us from introducing 
ipipe_timer_program(u64 tsc), assuming that such value must be based on 
our hrclock.

> Otherwise, we can pass a cpumask to ipipe_request_timers to tell on what
> cpus we need timer.
>

As I said: this is NOT a structural change. The problem statement is 
plain trivial, and we can address it, by evolving the I-pipe interface 
for more flexibility in handling clock devices, which in turn should 
remove such burden from the Xenomai core. The very initial

The point we are discussing is about whether using different clock 
devices to decouple the linux timekeeping machinery from the Xenomai 
timer subsystem is in essence possible. We seem to agree that this is 
possible without going back to the drawing board, which is the good news 
of the day.

> If we do not fix that one way or another, anyone wanting to use another
> timer than the default ones will have to provide a timer for each cpu core.
>

Incidentally, I'm moving the pipeline core to 3.4, so this may be a good 
opportunity to update the timer interface we expose to client domains as 
well.

-- 
Philippe.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/23/2012 12:02 PM, Philippe Gerum wrote:>> If we want to change
that, we have to fix Xenomai, which supposes there

>> is only one "timer_freq" (rthal_tunables.timer_freq). We can probably
>> get away with setting RTHAL_TIMER_FREQ to a per_cpu variable on all
>> architectures but arm which uses the frequency to pre-compute a
>> multiplicand to avoid division when converting between cpu frequency and
>> timer frequency in xnarch_program_timer_shot.
>>
> 
> Maybe it's time to move the hrtimer-freq <-> hrclock-freq computations 
> to some globally visible helpers into the pipeline core. Then, the 
> arch-dep sections of the pipeline could go wild doing these computations 
> the way they want to, without affecting the interface anymore. 
> Typically, there is nothing which would prevent us from introducing 
> ipipe_timer_program(u64 tsc), assuming that such value must be based on 
> our hrclock.


It means moving the asm/arith.h code to I-pipe, but since we need it to
support older patches, this means a whole new source of code duplication.

If you mean passing an absolute date to program the timer, I had thought
about it, I am not sure this is the right way to go:

- most ipipe timers use the clock_event callback anyway, which use a
delay as argument.
- it has problems of its own, basically, you have to do what all
free-running counter + match register do: re-read the free-running
counter after programming the match register to check that the
programmed date is not alread passed.
- On hardware timers with free-running counter and match register, this
avoids the problem of taking into account the timer calibration, but the
problem remains for decrementers (the most common case, anyway), and it
means that the calibration code would be moved to the I-pipe patch too,
and we have the code duplication issue again.

> 
>> Otherwise, we can pass a cpumask to ipipe_request_timers to tell on what
>> cpus we need timer.
>>
> 
> As I said: this is NOT a structural change. The problem statement is 
> plain trivial, and we can address it, by evolving the I-pipe interface 
> for more flexibility in handling clock devices, which in turn should 
> remove such burden from the Xenomai core. The very initial
> 
> The point we are discussing is about whether using different clock 
> devices to decouple the linux timekeeping machinery from the Xenomai 
> timer subsystem is in essence possible. We seem to agree that this is 
> possible without going back to the drawing board, which is the good news 
> of the day.
> 
>> If we do not fix that one way or another, anyone wanting to use another
>> timer than the default ones will have to provide a timer for each cpu core.
>>
> 
> Incidentally, I'm moving the pipeline core to 3.4, so this may be a good 
> opportunity to update the timer interface we expose to client domains as 
> well.


I would be in favor of adding the cpumask parameter to
ipipe_request_timers: it means that the "all timers with the same
frequency" issue will be restricted to cpus which xenomai wants to use.
And we have an issue when using the "supported_cpus" feature.

As I said, ipipe_request_timers will grab all the timers and switch them
to one-shot mode. However, Xenomai will only take control of some of them.

So, if the timers were running in periodic mode when
ipipe_request_timers was called (so, if CONFIG_HIGH_RES_TIMERS is not
enabled), and xenomai does not take control of them, it means the timer
will not be reprogrammed after the first tick: we just stopped the
periodic timer on the cpus xenomai will not use. Admittedly, this is a
corner case, but still, it exists.

There is a possible fix without breaking the interface, but we still get
the silly common frequency requirement.

I am also working on the core-3.4 branch right now.

-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Philippe Gerum]

On 07/23/2012 12:41 PM, Gilles Chanteperdrix wrote:
> On 07/23/2012 12:02 PM, Philippe Gerum wrote:>> If we want to change
> that, we have to fix Xenomai, which supposes there
>
>>> is only one "timer_freq" (rthal_tunables.timer_freq). We can probably
>>> get away with setting RTHAL_TIMER_FREQ to a per_cpu variable on all
>>> architectures but arm which uses the frequency to pre-compute a
>>> multiplicand to avoid division when converting between cpu frequency and
>>> timer frequency in xnarch_program_timer_shot.
>>>
>>
>> Maybe it's time to move the hrtimer-freq <-> hrclock-freq computations
>> to some globally visible helpers into the pipeline core. Then, the
>> arch-dep sections of the pipeline could go wild doing these computations
>> the way they want to, without affecting the interface anymore.
>> Typically, there is nothing which would prevent us from introducing
>> ipipe_timer_program(u64 tsc), assuming that such value must be based on
>> our hrclock.
>
>
> It means moving the asm/arith.h code to I-pipe, but since we need it to
> support older patches, this means a whole new source of code duplication.

This is clearly 3.x material, not 2.x since we have entered the 
maintenance cycle for 2.6.x, and we won't go for 2.7.x. We have delayed 
(my bad) 3.x for way too long.

>
> If you mean passing an absolute date to program the timer, I had thought
> about it, I am not sure this is the right way to go:
>

It's not about passing an absolute date, it is about passing a delay 
based on the hrclock to avoid the conversion step, and therefore make 
the requirement to access frequencies useless.

> - most ipipe timers use the clock_event callback anyway, which use a
> delay as argument.
> - it has problems of its own, basically, you have to do what all
> free-running counter + match register do: re-read the free-running
> counter after programming the match register to check that the
> programmed date is not alread passed.
> - On hardware timers with free-running counter and match register, this
> avoids the problem of taking into account the timer calibration, but the
> problem remains for decrementers (the most common case, anyway), and it
> means that the calibration code would be moved to the I-pipe patch too,
> and we have the code duplication issue again.
>

What I mean is to move all timer device-specific handling out of the 
Xenomai core, to the pipeline, where these things already live. The 
motivation behind the new pipeline core is to become Xenomai friendly, 
at last.

>>
>>> Otherwise, we can pass a cpumask to ipipe_request_timers to tell on what
>>> cpus we need timer.
>>>
>>
>> As I said: this is NOT a structural change. The problem statement is
>> plain trivial, and we can address it, by evolving the I-pipe interface
>> for more flexibility in handling clock devices, which in turn should
>> remove such burden from the Xenomai core. The very initial
>>
>> The point we are discussing is about whether using different clock
>> devices to decouple the linux timekeeping machinery from the Xenomai
>> timer subsystem is in essence possible. We seem to agree that this is
>> possible without going back to the drawing board, which is the good news
>> of the day.
>>
>>> If we do not fix that one way or another, anyone wanting to use another
>>> timer than the default ones will have to provide a timer for each cpu core.
>>>
>>
>> Incidentally, I'm moving the pipeline core to 3.4, so this may be a good
>> opportunity to update the timer interface we expose to client domains as
>> well.
>
>
> I would be in favor of adding the cpumask parameter to
> ipipe_request_timers: it means that the "all timers with the same
> frequency" issue will be restricted to cpus which xenomai wants to use.
> And we have an issue when using the "supported_cpus" feature.
>
> As I said, ipipe_request_timers will grab all the timers and switch them
> to one-shot mode. However, Xenomai will only take control of some of them.
>
> So, if the timers were running in periodic mode when
> ipipe_request_timers was called (so, if CONFIG_HIGH_RES_TIMERS is not
> enabled), and xenomai does not take control of them, it means the timer
> will not be reprogrammed after the first tick: we just stopped the
> periodic timer on the cpus xenomai will not use. Admittedly, this is a
> corner case, but still, it exists.
>
> There is a possible fix without breaking the interface, but we still get
> the silly common frequency requirement.
>

3.x, really. Let's break it.

> I am also working on the core-3.4 branch right now.
>

Do you have fixes to the generic core pending? If so, I'd like to merge 
them, so that I can eventually send out an updated patch to the blackfin 
folks.

-- 
Philippe.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/23/2012 12:57 PM, Philippe Gerum wrote:

>> There is a possible fix without breaking the interface, but we still get
>> the silly common frequency requirement.
>>
> 
> 3.x, really. Let's break it.


Ok. Will work on that. We do not really need asm/arith.h to implement
ipipe_timer_set the way you see it, if we restrict to 32 bits delays (as
we currently do in Xenomai), and this can be written without support
from arch-dep code.


> 
>> I am also working on the core-3.4 branch right now.
>>
> 
> Do you have fixes to the generic core pending? If so, I'd like to merge 
> them, so that I can eventually send out an updated patch to the blackfin 
> folks.


I have modified a bit the genpatches.sh script, but I do not think this
goes outside the I-pipe git.

-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/23/2012 12:57 PM, Philippe Gerum wrote:

> Do you have fixes to the generic core pending? If so, I'd like to merge 
> them, so that I can eventually send out an updated patch to the blackfin 
> folks.


I just pushed my branch (for-core-3.4 in the usual repository), I made 
two generic modifications:
- add a new raw_spin_lock_irqsave_cond service (and the converse unlock)
http://git.xenomai.org/ipipe-gch.git/?p=ipipe-gch.git;a=commit;h=70bf66b5dcb1b2a2a02b5b9aa7021880e93704e7
- iron the "generic irq chip" spinlock
http://git.xenomai.org/ipipe-gch.git/?p=ipipe-gch.git;a=commit;h=eeb81d4da57b4537f74c4b60785faef4f2942e2f


-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/24/2012 05:20 PM, Gilles Chanteperdrix wrote:

> On 07/23/2012 12:57 PM, Philippe Gerum wrote:
> 
>> Do you have fixes to the generic core pending? If so, I'd like to merge 
>> them, so that I can eventually send out an updated patch to the blackfin 
>> folks.
> 
> 
> I just pushed my branch (for-core-3.4 in the usual repository), I made 


It includes the work for the ARM architecture, but not yet x86

-- 
                                                                Gilles.

Re: [Xenomai] RT_TASK affinity on more than 8 CPU's

[Gilles Chanteperdrix]

On 07/24/2012 05:20 PM, Gilles Chanteperdrix wrote:

> On 07/24/2012 05:20 PM, Gilles Chanteperdrix wrote:
> 
>> On 07/23/2012 12:57 PM, Philippe Gerum wrote:
>>
>>> Do you have fixes to the generic core pending? If so, I'd like to merge 
>>> them, so that I can eventually send out an updated patch to the blackfin 
>>> folks.
>>
>>
>> I just pushed my branch (for-core-3.4 in the usual repository), I made 
> 
> 
> It includes the work for the ARM architecture, but not yet x86
> 

I have pushed the x86, as my atom survived a "xeno-regression-test" in
64 bits and 32 bits modes, it is time for others to test the patch.

-- 
                                                                Gilles.

[Xenomai] [I-PIPE] 3.4 update

[Philippe Gerum]

On 07/25/2012 11:48 AM, Gilles Chanteperdrix wrote:
> On 07/24/2012 05:20 PM, Gilles Chanteperdrix wrote:
>
>> On 07/24/2012 05:20 PM, Gilles Chanteperdrix wrote:
>>
>>> On 07/23/2012 12:57 PM, Philippe Gerum wrote:
>>>
>>>> Do you have fixes to the generic core pending? If so, I'd like to merge
>>>> them, so that I can eventually send out an updated patch to the blackfin
>>>> folks.
>>>
>>>
>>> I just pushed my branch (for-core-3.4 in the usual repository), I made
>>
>>
>> It includes the work for the ARM architecture, but not yet x86
>>
>
> I have pushed the x86, as my atom survived a "xeno-regression-test" in
> 64 bits and 32 bits modes, it is time for others to test the patch.
>

Ok, thanks. Blackfin looks ok here too, like ppc32 but I still have to 
validate on a bunch of platforms (85xx, 40x, 44x typically). ppc64 is 
being heavily refactored to simplify the pipeline core there as well, 
I've been spending way too much time upgrading for this arch in the 
past, this has to stop.

I'll take your note as a pull request and merge your bits asap. Thanks.

-- 
Philippe.