[Xenomai] Default options in the debian package

[Xenomai] Default options in the debian package

[Leopold Palomo-Avellaneda]

Hi,


in the last days we have had a thread in the orocos-list [1] about the 
convenience to have activated the option (enable-dlopen-skins) in xenomai to 
use it with orocos properly.

I don't know exactly what does it means. The configure says:

--enable-dlopen-skins   Disable TLS features and automatic main thread
                          mapping by the POSIX skin to allows dlopen'ing
                          Xenomai libs. [default=no]


I would like to know the implications of have activated this option by default 
in the debian package (at least) . It could be possible? Implies to many 
things?

Regards,


Leo



[1] 
http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html

-- 
--
Linux User 152692
Catalonia

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-19 13:46, Leopold Palomo-Avellaneda wrote:
> Hi,
> 
> 
> in the last days we have had a thread in the orocos-list [1] about the 
> convenience to have activated the option (enable-dlopen-skins) in xenomai to 
> use it with orocos properly.
> 
> I don't know exactly what does it means. The configure says:
> 
> --enable-dlopen-skins   Disable TLS features and automatic main thread
>                           mapping by the POSIX skin to allows dlopen'ing
>                           Xenomai libs. [default=no]
> 
> 
> I would like to know the implications of have activated this option by default 
> in the debian package (at least) . It could be possible? Implies to many 
> things?

Well, setting that feature extends the number of use cases for which the
debian package can be used as-is, namely loading skin libraries or other
libraries that require a skin lib via dlopen. We depend on this feature
here, thus carry a local patch against the debian rules, but so far I
thought that this is far from being a common requirement. Does Orocos
have a dynamic plugin concept?

The downside of make the feature default is a micro-performance
degradation (likely not an issue) and the lacking auto-shadowing of the
main thread when using POSIX skins. The latter is a behavioral change
that existing applications may depend and, thus, notice when updating to
the next Xenomai package version.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Default options in the debian package

[Gilles Chanteperdrix]

On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:

> [1] 
> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html


Hi,

that link does not tell us why you need this option. And that would be
the most important information.

If what you need to disable is TLS, then configuring xenomai with
--without-__thread is sufficient

If what you need is to avoid the main thread shadowing, we are not going
to configure xenomai with --enable-dlopen-skins as it breaks otherwise
conformant applications, but we can add an environment variable like
XENO_PTHREAD_NO_AUTO_SHADOW to allow supporting both situations.

-- 
                                                                Gilles.

Re: [Xenomai] Default options in the debian package

[Michael Haberler]

Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:

> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
> 
>> [1] 
>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
> 
> 
> Hi,
> 
> that link does not tell us why you need this option. And that would be
> the most important information.

with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly

- Michael

> 
> If what you need to disable is TLS, then configuring xenomai with
> --without-__thread is sufficient
> 
> If what you need is to avoid the main thread shadowing, we are not going
> to configure xenomai with --enable-dlopen-skins as it breaks otherwise
> conformant applications, but we can add an environment variable like
> XENO_PTHREAD_NO_AUTO_SHADOW to allow supporting both situations.
> 
> -- 
>                                                                Gilles.
> 
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-20 08:04, Michael Haberler wrote:
> 
> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
> 
>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>
>>> [1] 
>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>
>>
>> Hi,
>>
>> that link does not tell us why you need this option. And that would be
>> the most important information.
> 
> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly

OK, it looks like we should try harder to detect dlopen scenarios during
runtime to avoid build-time switches. This is likely Xenomai 3 material:

 - We need to disable TLS optimizations by default (no big deal).

 - In the POSIX skin constructor, we need to read out the mlockall
   state, lock everything if necessary, and restore the state
   accordingly afterward. The Nucleus may help us here if there is no
   adequate libc service (ABI change -> Xenomai 3).

 - IIRC, the problem with unconditional auto-shadowing back then were
   the improper scheduling parameters that POSIX used to apply. That
   was fixed a while back. So if we simple re-apply the current
   parameters, it should cause no harm in a dlopen scenario. But I need
   to check this again at work against our scenario.

Jan


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20130420/d3c1b2e3/attachment.pgp>

Re: [Xenomai] Default options in the debian package

[Gilles Chanteperdrix]

On 04/20/2013 10:19 AM, Jan Kiszka wrote:

> On 2013-04-20 08:04, Michael Haberler wrote:
>>
>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>
>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>
>>>> [1] 
>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>
>>>
>>> Hi,
>>>
>>> that link does not tell us why you need this option. And that would be
>>> the most important information.
>>
>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
> 
> OK, it looks like we should try harder to detect dlopen scenarios during
> runtime to avoid build-time switches. This is likely Xenomai 3 material:
> 
>  - We need to disable TLS optimizations by default (no big deal).
> 
>  - In the POSIX skin constructor, we need to read out the mlockall
>    state, lock everything if necessary, and restore the state
>    accordingly afterward. The Nucleus may help us here if there is no
>    adequate libc service (ABI change -> Xenomai 3).
> 
>  - IIRC, the problem with unconditional auto-shadowing back then were
>    the improper scheduling parameters that POSIX used to apply. That
>    was fixed a while back. So if we simple re-apply the current
>    parameters, it should cause no harm in a dlopen scenario. But I need
>    to check this again at work against our scenario.


You do not like the idea of an environment variable allowing to disable
the automatic shadowing?

-- 
                                                                Gilles.

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
> 
>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>
>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>
>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>
>>>>> [1] 
>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>
>>>>
>>>> Hi,
>>>>
>>>> that link does not tell us why you need this option. And that would be
>>>> the most important information.
>>>
>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>
>> OK, it looks like we should try harder to detect dlopen scenarios during
>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>
>>  - We need to disable TLS optimizations by default (no big deal).
>>
>>  - In the POSIX skin constructor, we need to read out the mlockall
>>    state, lock everything if necessary, and restore the state
>>    accordingly afterward. The Nucleus may help us here if there is no
>>    adequate libc service (ABI change -> Xenomai 3).
>>
>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>    the improper scheduling parameters that POSIX used to apply. That
>>    was fixed a while back. So if we simple re-apply the current
>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>    to check this again at work against our scenario.
> 
> 
> You do not like the idea of an environment variable allowing to disable
> the automatic shadowing?

Not as a long-term solution as it is user-unfriendly. But it can be an
option worth considering for 2.6.

Jan


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20130420/27f3169f/attachment.pgp>

Re: [Xenomai] Default options in the debian package

[Gilles Chanteperdrix]

On 04/20/2013 05:18 PM, Jan Kiszka wrote:

> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>
>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>
>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>
>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>
>>>>>> [1] 
>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> that link does not tell us why you need this option. And that would be
>>>>> the most important information.
>>>>
>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>
>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>
>>>  - We need to disable TLS optimizations by default (no big deal).
>>>
>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>    state, lock everything if necessary, and restore the state
>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>    adequate libc service (ABI change -> Xenomai 3).
>>>
>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>    the improper scheduling parameters that POSIX used to apply. That
>>>    was fixed a while back. So if we simple re-apply the current
>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>    to check this again at work against our scenario.
>>
>>
>> You do not like the idea of an environment variable allowing to disable
>> the automatic shadowing?
> 
> Not as a long-term solution as it is user-unfriendly. But it can be an
> option worth considering for 2.6.


Apparently -forge is already doing it. The advantage of this solution is
that the same binary serves well several usages, if we intend to provide
packages as generic as possible, this seems like the way to go. Several
of the changes I made in the last few weeks go in the same direction.


-- 
                                                                Gilles.

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
> 
>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>
>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>
>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>
>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>
>>>>>>> [1] 
>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> that link does not tell us why you need this option. And that would be
>>>>>> the most important information.
>>>>>
>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>
>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>
>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>
>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>    state, lock everything if necessary, and restore the state
>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>
>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>    was fixed a while back. So if we simple re-apply the current
>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>    to check this again at work against our scenario.
>>>
>>>
>>> You do not like the idea of an environment variable allowing to disable
>>> the automatic shadowing?
>>
>> Not as a long-term solution as it is user-unfriendly. But it can be an
>> option worth considering for 2.6.
> 
> 
> Apparently -forge is already doing it. The advantage of this solution is
> that the same binary serves well several usages, if we intend to provide
> packages as generic as possible, this seems like the way to go. Several
> of the changes I made in the last few weeks go in the same direction.

I'm not sure if there is added value for the controlling auto-shadowing
in general. But for the case it is in conflict with dlopen, the solution
I'm proposing is clearly superior as it removes those conflicts
automatically.

Of course, an environment variable control can exist in parallel if
there is a need beyond the dlopen conflict resolution.

Jan


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20130420/b4fb6bd6/attachment.pgp>

Re: [Xenomai] Default options in the debian package

[Gilles Chanteperdrix]

On 04/20/2013 05:27 PM, Jan Kiszka wrote:

> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>
>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>
>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>
>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>
>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>
>>>>>>>> [1] 
>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>> the most important information.
>>>>>>
>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>
>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>
>>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>>
>>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>>    state, lock everything if necessary, and restore the state
>>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>>
>>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>>    was fixed a while back. So if we simple re-apply the current
>>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>    to check this again at work against our scenario.
>>>>
>>>>
>>>> You do not like the idea of an environment variable allowing to disable
>>>> the automatic shadowing?
>>>
>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>> option worth considering for 2.6.
>>
>>
>> Apparently -forge is already doing it. The advantage of this solution is
>> that the same binary serves well several usages, if we intend to provide
>> packages as generic as possible, this seems like the way to go. Several
>> of the changes I made in the last few weeks go in the same direction.
> 
> I'm not sure if there is added value for the controlling auto-shadowing
> in general. But for the case it is in conflict with dlopen, the solution
> I'm proposing is clearly superior as it removes those conflicts
> automatically.
> 
> Of course, an environment variable control can exist in parallel if
> there is a need beyond the dlopen conflict resolution.


The difference with what you propose is that you propose a syscall to
get the mlockall state. Another solution would be not to call munlockall
after the main thread shadowing, this looks less complicated and does
not require ABI changes.


-- 
                                                                Gilles.

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-20 17:30, Gilles Chanteperdrix wrote:
> On 04/20/2013 05:27 PM, Jan Kiszka wrote:
> 
>> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>>
>>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>>
>>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>>
>>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>>
>>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>>
>>>>>>>>> [1] 
>>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>>> the most important information.
>>>>>>>
>>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>>
>>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>>
>>>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>>>
>>>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>>>    state, lock everything if necessary, and restore the state
>>>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>>>
>>>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>>>    was fixed a while back. So if we simple re-apply the current
>>>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>>    to check this again at work against our scenario.
>>>>>
>>>>>
>>>>> You do not like the idea of an environment variable allowing to disable
>>>>> the automatic shadowing?
>>>>
>>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>>> option worth considering for 2.6.
>>>
>>>
>>> Apparently -forge is already doing it. The advantage of this solution is
>>> that the same binary serves well several usages, if we intend to provide
>>> packages as generic as possible, this seems like the way to go. Several
>>> of the changes I made in the last few weeks go in the same direction.
>>
>> I'm not sure if there is added value for the controlling auto-shadowing
>> in general. But for the case it is in conflict with dlopen, the solution
>> I'm proposing is clearly superior as it removes those conflicts
>> automatically.
>>
>> Of course, an environment variable control can exist in parallel if
>> there is a need beyond the dlopen conflict resolution.
> 
> 
> The difference with what you propose is that you propose a syscall to
> get the mlockall state. Another solution would be not to call munlockall
> after the main thread shadowing, this looks less complicated and does
> not require ABI changes.

Yes, that's an option as well. But then we should apply this
consistently, invoking mlockall from all skin init functions
unconditionally. The nucleus depends on this anyway. Not sure if such
change would be fine for 2.6 - you decide.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Default options in the debian package

[Leopold Palomo-Avellaneda]

A Divendres, 19 d'abril de 2013, Gilles Chanteperdrix va escriure:
> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
> 
> > [1] 
> > http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-
April/006986.html
> 
> 
> Hi,
> 
> that link does not tell us why you need this option. And that would be
> the most important information.

:-)

you are right. I have pointed vague information about that. The complete 
thread commented that some people are getting errors using orocos with 
xenomai. Orocos use intensively the plugin architecture. As you can see in 
this thread:

http://www.orocos.org/forum/orocos/orocos-users/problem-orocos-under-xenomai

some user point that the solution to load the plugin is to activate this 
option in xenomai libs.

> If what you need to disable is TLS, then configuring xenomai with
> --without-__thread is sufficient
> 
> If what you need is to avoid the main thread shadowing, we are not going
> to configure xenomai with --enable-dlopen-skins as it breaks otherwise
> conformant applications, but we can add an environment variable like
> XENO_PTHREAD_NO_AUTO_SHADOW to allow supporting both situations.
> 

I don't know if this is a solution, it overpass my knowledge. I would like to 
put the detail that, under my subjective point of view, without numerical 
data, the orocos users (and cnc users) are the main users of the debian 
package. If, we activate that option and we solve this issue, no package 
recompilation are needed, that's all. 

I don't know the deep repercussion of that decision. Jan Kiszka said in 
another mail:

> default is a micro-performance degradation (likely not an issue) and the 
lacking auto-shadowing of the main thread when using POSIX skins

You are the main developers, your decisions are the most important, but I 
think that a lot of people activate that, at least the users that use that 
debian package.

Best regards,

Leo

-- 
--
Linux User 152692
Catalonia

Re: [Xenomai] Default options in the debian package

[Gilles Chanteperdrix]

On 04/22/2013 09:11 AM, Jan Kiszka wrote:

> On 2013-04-20 17:30, Gilles Chanteperdrix wrote:
>> On 04/20/2013 05:27 PM, Jan Kiszka wrote:
>>
>>> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>>>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>>>
>>>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>>>
>>>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>>>
>>>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>>>
>>>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>>>
>>>>>>>>>> [1] 
>>>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>>>> the most important information.
>>>>>>>>
>>>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>>>
>>>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>>>
>>>>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>>>>
>>>>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>>>>    state, lock everything if necessary, and restore the state
>>>>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>>>>
>>>>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>>>>    was fixed a while back. So if we simple re-apply the current
>>>>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>>>    to check this again at work against our scenario.
>>>>>>
>>>>>>
>>>>>> You do not like the idea of an environment variable allowing to disable
>>>>>> the automatic shadowing?
>>>>>
>>>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>>>> option worth considering for 2.6.
>>>>
>>>>
>>>> Apparently -forge is already doing it. The advantage of this solution is
>>>> that the same binary serves well several usages, if we intend to provide
>>>> packages as generic as possible, this seems like the way to go. Several
>>>> of the changes I made in the last few weeks go in the same direction.
>>>
>>> I'm not sure if there is added value for the controlling auto-shadowing
>>> in general. But for the case it is in conflict with dlopen, the solution
>>> I'm proposing is clearly superior as it removes those conflicts
>>> automatically.
>>>
>>> Of course, an environment variable control can exist in parallel if
>>> there is a need beyond the dlopen conflict resolution.
>>
>>
>> The difference with what you propose is that you propose a syscall to
>> get the mlockall state. Another solution would be not to call munlockall
>> after the main thread shadowing, this looks less complicated and does
>> not require ABI changes.
> 
> Yes, that's an option as well. But then we should apply this
> consistently, invoking mlockall from all skin init functions
> unconditionally. The nucleus depends on this anyway. Not sure if such
> change would be fine for 2.6 - you decide.


What we could do is:
- if XENO_NOSHADOW is set, shadow the main thread, and call mlockall
- if it is not set, do not shadow the main thread or call
mlockall/munlockall


-- 
                                                                Gilles.

Re: [Xenomai] Default options in the debian package

[Gilles Chanteperdrix]

On 04/22/2013 09:18 AM, Leopold Palomo-Avellaneda wrote:

> A Divendres, 19 d'abril de 2013, Gilles Chanteperdrix va escriure:
>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>
>>> [1] 
>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-
> April/006986.html
>>
>>
>> Hi,
>>
>> that link does not tell us why you need this option. And that would be
>> the most important information.
> 
> :-)
> 
> you are right. I have pointed vague information about that. The complete 
> thread commented that some people are getting errors using orocos with 
> xenomai. Orocos use intensively the plugin architecture. As you can see in 
> this thread:
> 
> http://www.orocos.org/forum/orocos/orocos-users/problem-orocos-under-xenomai
> 
> some user point that the solution to load the plugin is to activate this 
> option in xenomai libs.
> 
>> If what you need to disable is TLS, then configuring xenomai with
>> --without-__thread is sufficient
>>
>> If what you need is to avoid the main thread shadowing, we are not going
>> to configure xenomai with --enable-dlopen-skins as it breaks otherwise
>> conformant applications, but we can add an environment variable like
>> XENO_PTHREAD_NO_AUTO_SHADOW to allow supporting both situations.
>>
> 
> I don't know if this is a solution, it overpass my knowledge. I would like to 
> put the detail that, under my subjective point of view, without numerical 
> data, the orocos users (and cnc users) are the main users of the debian 
> package. If, we activate that option and we solve this issue, no package 
> recompilation are needed, that's all. 
> 
> I don't know the deep repercussion of that decision. Jan Kiszka said in 
> another mail:
> 
>> default is a micro-performance degradation (likely not an issue) and the 
> lacking auto-shadowing of the main thread when using POSIX skins
> 
> You are the main developers, your decisions are the most important, but I 
> think that a lot of people activate that, at least the users that use that 
> debian package.


The problem is that if we do not shadow the main thread, we break
programs that used to work, and also break programs for reasons which
are hard to understand for a new-comer. Hence the question, would not
--without__thread be sufficient for Orocos? Only an Orocos user has the
answer to this question.

-- 
                                                                Gilles.

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-22 13:37, Gilles Chanteperdrix wrote:
> On 04/22/2013 09:11 AM, Jan Kiszka wrote:
> 
>> On 2013-04-20 17:30, Gilles Chanteperdrix wrote:
>>> On 04/20/2013 05:27 PM, Jan Kiszka wrote:
>>>
>>>> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>>>>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>>>>
>>>>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>>>>
>>>>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>>>>
>>>>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>>>>
>>>>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>>>>
>>>>>>>>>>> [1] 
>>>>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>>>>> the most important information.
>>>>>>>>>
>>>>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>>>>
>>>>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>>>>
>>>>>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>>>>>
>>>>>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>>>>>    state, lock everything if necessary, and restore the state
>>>>>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>>>>>
>>>>>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>>>>>    was fixed a while back. So if we simple re-apply the current
>>>>>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>>>>    to check this again at work against our scenario.
>>>>>>>
>>>>>>>
>>>>>>> You do not like the idea of an environment variable allowing to disable
>>>>>>> the automatic shadowing?
>>>>>>
>>>>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>>>>> option worth considering for 2.6.
>>>>>
>>>>>
>>>>> Apparently -forge is already doing it. The advantage of this solution is
>>>>> that the same binary serves well several usages, if we intend to provide
>>>>> packages as generic as possible, this seems like the way to go. Several
>>>>> of the changes I made in the last few weeks go in the same direction.
>>>>
>>>> I'm not sure if there is added value for the controlling auto-shadowing
>>>> in general. But for the case it is in conflict with dlopen, the solution
>>>> I'm proposing is clearly superior as it removes those conflicts
>>>> automatically.
>>>>
>>>> Of course, an environment variable control can exist in parallel if
>>>> there is a need beyond the dlopen conflict resolution.
>>>
>>>
>>> The difference with what you propose is that you propose a syscall to
>>> get the mlockall state. Another solution would be not to call munlockall
>>> after the main thread shadowing, this looks less complicated and does
>>> not require ABI changes.
>>
>> Yes, that's an option as well. But then we should apply this
>> consistently, invoking mlockall from all skin init functions
>> unconditionally. The nucleus depends on this anyway. Not sure if such
>> change would be fine for 2.6 - you decide.
> 
> 
> What we could do is:
> - if XENO_NOSHADOW is set, shadow the main thread, and call mlockall
> - if it is not set, do not shadow the main thread or call
> mlockall/munlockall

That's not what I suggested. I was questioning the value of _not_ doing
mlockall automatically during init, thus reducing user duties. That
would reduce the need to think about XENO_NOSHADOW or not as a normal user.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Default options in the debian package

[Leopold Palomo-Avellaneda]

A Dilluns, 22 d'abril de 2013, Gilles Chanteperdrix va escriure:
> On 04/22/2013 09:18 AM, Leopold Palomo-Avellaneda wrote:
> 
> > A Divendres, 19 d'abril de 2013, Gilles Chanteperdrix va escriure:
> >> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
> >>
> >>> [1] 
> >>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-
> > April/006986.html
> >>
> >>
> >> Hi,
> >>
> >> that link does not tell us why you need this option. And that would be
> >> the most important information.
> > 
> > :-)
> > 
> > you are right. I have pointed vague information about that. The complete 
> > thread commented that some people are getting errors using orocos with 
> > xenomai. Orocos use intensively the plugin architecture. As you can see in 
> > this thread:
> > 
> > http://www.orocos.org/forum/orocos/orocos-users/problem-orocos-under-
xenomai
> > 
> > some user point that the solution to load the plugin is to activate this 
> > option in xenomai libs.
> > 
> >> If what you need to disable is TLS, then configuring xenomai with
> >> --without-__thread is sufficient
> >>
> >> If what you need is to avoid the main thread shadowing, we are not going
> >> to configure xenomai with --enable-dlopen-skins as it breaks otherwise
> >> conformant applications, but we can add an environment variable like
> >> XENO_PTHREAD_NO_AUTO_SHADOW to allow supporting both situations.
> >>
> > 
> > I don't know if this is a solution, it overpass my knowledge. I would like 
to 
> > put the detail that, under my subjective point of view, without numerical 
> > data, the orocos users (and cnc users) are the main users of the debian 
> > package. If, we activate that option and we solve this issue, no package 
> > recompilation are needed, that's all. 
> > 
> > I don't know the deep repercussion of that decision. Jan Kiszka said in 
> > another mail:
> > 
> >> default is a micro-performance degradation (likely not an issue) and the 
> > lacking auto-shadowing of the main thread when using POSIX skins
> > 
> > You are the main developers, your decisions are the most important, but I 
> > think that a lot of people activate that, at least the users that use that 
> > debian package.
> 
> 
> The problem is that if we do not shadow the main thread, we break
> programs that used to work, and also break programs for reasons which
> are hard to understand for a new-comer. Hence the question, would not
> --without__thread be sufficient for Orocos? Only an Orocos user has the
> answer to this question.
> 

I don't know. Orocos use intensively the plugins architecture so I don't know 
if this could solve it. But, as another person in the list has pointed, if you 
would like to use the Python modules you need it too.

I have asked to the Orocos main developer and I would try to test it,

Regards,

Leo


-- 
--
Linux User 152692
Catalonia

Re: [Xenomai] Default options in the debian package

[Jeff Webb]

On 04/20/2013 01:04 AM, Michael Haberler wrote:
>
> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>
>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>
>>> [1]
>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>
>>
>> Hi,
>>
>> that link does not tell us why you need this option. And that would be
>> the most important information.
>
> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
  
For what it's worth, we use --enable-dlopen-skins (and build Python modules) for some of our applications as well.

-Jeff

Re: [Xenomai] Default options in the debian package

[Gilles Chanteperdrix]

On 04/22/2013 03:42 PM, Jan Kiszka wrote:

> On 2013-04-22 13:37, Gilles Chanteperdrix wrote:
>> On 04/22/2013 09:11 AM, Jan Kiszka wrote:
>>
>>> On 2013-04-20 17:30, Gilles Chanteperdrix wrote:
>>>> On 04/20/2013 05:27 PM, Jan Kiszka wrote:
>>>>
>>>>> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>>>>>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>>>>>
>>>>>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>>>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>>>>>
>>>>>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>>>>>
>>>>>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>>>>>
>>>>>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>>>>>
>>>>>>>>>>>> [1] 
>>>>>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>>>>>> the most important information.
>>>>>>>>>>
>>>>>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>>>>>
>>>>>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>>>>>
>>>>>>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>>>>>>
>>>>>>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>>>>>>    state, lock everything if necessary, and restore the state
>>>>>>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>>>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>>>>>>
>>>>>>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>>>>>>    was fixed a while back. So if we simple re-apply the current
>>>>>>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>>>>>    to check this again at work against our scenario.
>>>>>>>>
>>>>>>>>
>>>>>>>> You do not like the idea of an environment variable allowing to disable
>>>>>>>> the automatic shadowing?
>>>>>>>
>>>>>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>>>>>> option worth considering for 2.6.
>>>>>>
>>>>>>
>>>>>> Apparently -forge is already doing it. The advantage of this solution is
>>>>>> that the same binary serves well several usages, if we intend to provide
>>>>>> packages as generic as possible, this seems like the way to go. Several
>>>>>> of the changes I made in the last few weeks go in the same direction.
>>>>>
>>>>> I'm not sure if there is added value for the controlling auto-shadowing
>>>>> in general. But for the case it is in conflict with dlopen, the solution
>>>>> I'm proposing is clearly superior as it removes those conflicts
>>>>> automatically.
>>>>>
>>>>> Of course, an environment variable control can exist in parallel if
>>>>> there is a need beyond the dlopen conflict resolution.
>>>>
>>>>
>>>> The difference with what you propose is that you propose a syscall to
>>>> get the mlockall state. Another solution would be not to call munlockall
>>>> after the main thread shadowing, this looks less complicated and does
>>>> not require ABI changes.
>>>
>>> Yes, that's an option as well. But then we should apply this
>>> consistently, invoking mlockall from all skin init functions
>>> unconditionally. The nucleus depends on this anyway. Not sure if such
>>> change would be fine for 2.6 - you decide.
>>
>>
>> What we could do is:
>> - if XENO_NOSHADOW is set, shadow the main thread, and call mlockall
>> - if it is not set, do not shadow the main thread or call
>> mlockall/munlockall
> 
> That's not what I suggested. I was questioning the value of _not_ doing
> mlockall automatically during init, thus reducing user duties. That
> would reduce the need to think about XENO_NOSHADOW or not as a normal user.


Currently, when the posix skin library start, it does:
mlockall
shadow main thread
munlockall

Now, the munlockall is certainly an issue when dlopening

What I propose instead is to do:
if (!getenv("XENO_NOSHADOW")) {
	mlockall
	shadow main thread
}

That will avoid the problem with calling munlockall, and if people who
currently use --enable-dlopen-skins really want to avoid shadowing the
main thread (which I doubt), they can set the environment variable
XENO_NOSHADOW.

-- 
                                                                Gilles.

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-22 20:57, Gilles Chanteperdrix wrote:
> On 04/22/2013 03:42 PM, Jan Kiszka wrote:
> 
>> On 2013-04-22 13:37, Gilles Chanteperdrix wrote:
>>> On 04/22/2013 09:11 AM, Jan Kiszka wrote:
>>>
>>>> On 2013-04-20 17:30, Gilles Chanteperdrix wrote:
>>>>> On 04/20/2013 05:27 PM, Jan Kiszka wrote:
>>>>>
>>>>>> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>>>>>>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>>>>>>
>>>>>>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>>>>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>>>>>>
>>>>>>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>>>>>>
>>>>>>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>>>>>>
>>>>>>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> [1] 
>>>>>>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>>>>>>> the most important information.
>>>>>>>>>>>
>>>>>>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>>>>>>
>>>>>>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>>>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>>>>>>
>>>>>>>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>>>>>>>
>>>>>>>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>>>>>>>    state, lock everything if necessary, and restore the state
>>>>>>>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>>>>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>>>>>>>
>>>>>>>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>>>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>>>>>>>    was fixed a while back. So if we simple re-apply the current
>>>>>>>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>>>>>>    to check this again at work against our scenario.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You do not like the idea of an environment variable allowing to disable
>>>>>>>>> the automatic shadowing?
>>>>>>>>
>>>>>>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>>>>>>> option worth considering for 2.6.
>>>>>>>
>>>>>>>
>>>>>>> Apparently -forge is already doing it. The advantage of this solution is
>>>>>>> that the same binary serves well several usages, if we intend to provide
>>>>>>> packages as generic as possible, this seems like the way to go. Several
>>>>>>> of the changes I made in the last few weeks go in the same direction.
>>>>>>
>>>>>> I'm not sure if there is added value for the controlling auto-shadowing
>>>>>> in general. But for the case it is in conflict with dlopen, the solution
>>>>>> I'm proposing is clearly superior as it removes those conflicts
>>>>>> automatically.
>>>>>>
>>>>>> Of course, an environment variable control can exist in parallel if
>>>>>> there is a need beyond the dlopen conflict resolution.
>>>>>
>>>>>
>>>>> The difference with what you propose is that you propose a syscall to
>>>>> get the mlockall state. Another solution would be not to call munlockall
>>>>> after the main thread shadowing, this looks less complicated and does
>>>>> not require ABI changes.
>>>>
>>>> Yes, that's an option as well. But then we should apply this
>>>> consistently, invoking mlockall from all skin init functions
>>>> unconditionally. The nucleus depends on this anyway. Not sure if such
>>>> change would be fine for 2.6 - you decide.
>>>
>>>
>>> What we could do is:
>>> - if XENO_NOSHADOW is set, shadow the main thread, and call mlockall
>>> - if it is not set, do not shadow the main thread or call
>>> mlockall/munlockall
>>
>> That's not what I suggested. I was questioning the value of _not_ doing
>> mlockall automatically during init, thus reducing user duties. That
>> would reduce the need to think about XENO_NOSHADOW or not as a normal user.
> 
> 
> Currently, when the posix skin library start, it does:
> mlockall
> shadow main thread
> munlockall
> 
> Now, the munlockall is certainly an issue when dlopening
> 
> What I propose instead is to do:
> if (!getenv("XENO_NOSHADOW")) {
> 	mlockall
> 	shadow main thread
> }
> 
> That will avoid the problem with calling munlockall, and if people who
> currently use --enable-dlopen-skins really want to avoid shadowing the
> main thread (which I doubt), they can set the environment variable
> XENO_NOSHADOW.

Totally clear. But why still requiring the application to call mlockall
if we do not autoshadow or use a different skin? It's pointless to have
this explicit call in the application if we can easily do this from the
library.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Default options in the debian package

[Gilles Chanteperdrix]

On 04/23/2013 10:21 AM, Jan Kiszka wrote:

> On 2013-04-22 20:57, Gilles Chanteperdrix wrote:
>> On 04/22/2013 03:42 PM, Jan Kiszka wrote:
>>
>>> On 2013-04-22 13:37, Gilles Chanteperdrix wrote:
>>>> On 04/22/2013 09:11 AM, Jan Kiszka wrote:
>>>>
>>>>> On 2013-04-20 17:30, Gilles Chanteperdrix wrote:
>>>>>> On 04/20/2013 05:27 PM, Jan Kiszka wrote:
>>>>>>
>>>>>>> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>>>>>>>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>>>>>>>
>>>>>>>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>>>>>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>>>>>>>
>>>>>>>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>>>>>>>
>>>>>>>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] 
>>>>>>>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>>>>>>>> the most important information.
>>>>>>>>>>>>
>>>>>>>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>>>>>>>
>>>>>>>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>>>>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>>>>>>>
>>>>>>>>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>>>>>>>>
>>>>>>>>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>>>>>>>>    state, lock everything if necessary, and restore the state
>>>>>>>>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>>>>>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>>>>>>>>
>>>>>>>>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>>>>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>>>>>>>>    was fixed a while back. So if we simple re-apply the current
>>>>>>>>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>>>>>>>    to check this again at work against our scenario.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You do not like the idea of an environment variable allowing to disable
>>>>>>>>>> the automatic shadowing?
>>>>>>>>>
>>>>>>>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>>>>>>>> option worth considering for 2.6.
>>>>>>>>
>>>>>>>>
>>>>>>>> Apparently -forge is already doing it. The advantage of this solution is
>>>>>>>> that the same binary serves well several usages, if we intend to provide
>>>>>>>> packages as generic as possible, this seems like the way to go. Several
>>>>>>>> of the changes I made in the last few weeks go in the same direction.
>>>>>>>
>>>>>>> I'm not sure if there is added value for the controlling auto-shadowing
>>>>>>> in general. But for the case it is in conflict with dlopen, the solution
>>>>>>> I'm proposing is clearly superior as it removes those conflicts
>>>>>>> automatically.
>>>>>>>
>>>>>>> Of course, an environment variable control can exist in parallel if
>>>>>>> there is a need beyond the dlopen conflict resolution.
>>>>>>
>>>>>>
>>>>>> The difference with what you propose is that you propose a syscall to
>>>>>> get the mlockall state. Another solution would be not to call munlockall
>>>>>> after the main thread shadowing, this looks less complicated and does
>>>>>> not require ABI changes.
>>>>>
>>>>> Yes, that's an option as well. But then we should apply this
>>>>> consistently, invoking mlockall from all skin init functions
>>>>> unconditionally. The nucleus depends on this anyway. Not sure if such
>>>>> change would be fine for 2.6 - you decide.
>>>>
>>>>
>>>> What we could do is:
>>>> - if XENO_NOSHADOW is set, shadow the main thread, and call mlockall
>>>> - if it is not set, do not shadow the main thread or call
>>>> mlockall/munlockall
>>>
>>> That's not what I suggested. I was questioning the value of _not_ doing
>>> mlockall automatically during init, thus reducing user duties. That
>>> would reduce the need to think about XENO_NOSHADOW or not as a normal user.
>>
>>
>> Currently, when the posix skin library start, it does:
>> mlockall
>> shadow main thread
>> munlockall
>>
>> Now, the munlockall is certainly an issue when dlopening
>>
>> What I propose instead is to do:
>> if (!getenv("XENO_NOSHADOW")) {
>> 	mlockall
>> 	shadow main thread
>> }
>>
>> That will avoid the problem with calling munlockall, and if people who
>> currently use --enable-dlopen-skins really want to avoid shadowing the
>> main thread (which I doubt), they can set the environment variable
>> XENO_NOSHADOW.
> 
> Totally clear. But why still requiring the application to call mlockall
> if we do not autoshadow or use a different skin? It's pointless to have
> this explicit call in the application if we can easily do this from the
> library.


Well, we have --enable-posix-auto-mlockall for that. But you are right,
we get rid of all this and enable automatic mlockall for all skins. I do
not see any drawbacks.


-- 
                                                                Gilles.

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-23 13:20, Gilles Chanteperdrix wrote:
> On 04/23/2013 10:21 AM, Jan Kiszka wrote:
> 
>> On 2013-04-22 20:57, Gilles Chanteperdrix wrote:
>>> On 04/22/2013 03:42 PM, Jan Kiszka wrote:
>>>
>>>> On 2013-04-22 13:37, Gilles Chanteperdrix wrote:
>>>>> On 04/22/2013 09:11 AM, Jan Kiszka wrote:
>>>>>
>>>>>> On 2013-04-20 17:30, Gilles Chanteperdrix wrote:
>>>>>>> On 04/20/2013 05:27 PM, Jan Kiszka wrote:
>>>>>>>
>>>>>>>> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>>>>>>>>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>>>>>>>>
>>>>>>>>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>>>>>>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] 
>>>>>>>>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>>>>>>>>> the most important information.
>>>>>>>>>>>>>
>>>>>>>>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>>>>>>>>
>>>>>>>>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>>>>>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>>>>>>>>
>>>>>>>>>>>>  - We need to disable TLS optimizations by default (no big deal).
>>>>>>>>>>>>
>>>>>>>>>>>>  - In the POSIX skin constructor, we need to read out the mlockall
>>>>>>>>>>>>    state, lock everything if necessary, and restore the state
>>>>>>>>>>>>    accordingly afterward. The Nucleus may help us here if there is no
>>>>>>>>>>>>    adequate libc service (ABI change -> Xenomai 3).
>>>>>>>>>>>>
>>>>>>>>>>>>  - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>>>>>>>>    the improper scheduling parameters that POSIX used to apply. That
>>>>>>>>>>>>    was fixed a while back. So if we simple re-apply the current
>>>>>>>>>>>>    parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>>>>>>>>    to check this again at work against our scenario.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> You do not like the idea of an environment variable allowing to disable
>>>>>>>>>>> the automatic shadowing?
>>>>>>>>>>
>>>>>>>>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>>>>>>>>> option worth considering for 2.6.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apparently -forge is already doing it. The advantage of this solution is
>>>>>>>>> that the same binary serves well several usages, if we intend to provide
>>>>>>>>> packages as generic as possible, this seems like the way to go. Several
>>>>>>>>> of the changes I made in the last few weeks go in the same direction.
>>>>>>>>
>>>>>>>> I'm not sure if there is added value for the controlling auto-shadowing
>>>>>>>> in general. But for the case it is in conflict with dlopen, the solution
>>>>>>>> I'm proposing is clearly superior as it removes those conflicts
>>>>>>>> automatically.
>>>>>>>>
>>>>>>>> Of course, an environment variable control can exist in parallel if
>>>>>>>> there is a need beyond the dlopen conflict resolution.
>>>>>>>
>>>>>>>
>>>>>>> The difference with what you propose is that you propose a syscall to
>>>>>>> get the mlockall state. Another solution would be not to call munlockall
>>>>>>> after the main thread shadowing, this looks less complicated and does
>>>>>>> not require ABI changes.
>>>>>>
>>>>>> Yes, that's an option as well. But then we should apply this
>>>>>> consistently, invoking mlockall from all skin init functions
>>>>>> unconditionally. The nucleus depends on this anyway. Not sure if such
>>>>>> change would be fine for 2.6 - you decide.
>>>>>
>>>>>
>>>>> What we could do is:
>>>>> - if XENO_NOSHADOW is set, shadow the main thread, and call mlockall
>>>>> - if it is not set, do not shadow the main thread or call
>>>>> mlockall/munlockall
>>>>
>>>> That's not what I suggested. I was questioning the value of _not_ doing
>>>> mlockall automatically during init, thus reducing user duties. That
>>>> would reduce the need to think about XENO_NOSHADOW or not as a normal user.
>>>
>>>
>>> Currently, when the posix skin library start, it does:
>>> mlockall
>>> shadow main thread
>>> munlockall
>>>
>>> Now, the munlockall is certainly an issue when dlopening
>>>
>>> What I propose instead is to do:
>>> if (!getenv("XENO_NOSHADOW")) {
>>> 	mlockall
>>> 	shadow main thread
>>> }
>>>
>>> That will avoid the problem with calling munlockall, and if people who
>>> currently use --enable-dlopen-skins really want to avoid shadowing the
>>> main thread (which I doubt), they can set the environment variable
>>> XENO_NOSHADOW.
>>
>> Totally clear. But why still requiring the application to call mlockall
>> if we do not autoshadow or use a different skin? It's pointless to have
>> this explicit call in the application if we can easily do this from the
>> library.
> 
> 
> Well, we have --enable-posix-auto-mlockall for that. But you are right,
> we get rid of all this and enable automatic mlockall for all skins. I do
> not see any drawbacks.

Perfect. Will work out some patches.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Default options in the debian package

[Philippe Gerum]

On 04/23/2013 01:20 PM, Gilles Chanteperdrix wrote:
> On 04/23/2013 10:21 AM, Jan Kiszka wrote:
>
>> On 2013-04-22 20:57, Gilles Chanteperdrix wrote:
>>> On 04/22/2013 03:42 PM, Jan Kiszka wrote:
>>>
>>>> On 2013-04-22 13:37, Gilles Chanteperdrix wrote:
>>>>> On 04/22/2013 09:11 AM, Jan Kiszka wrote:
>>>>>
>>>>>> On 2013-04-20 17:30, Gilles Chanteperdrix wrote:
>>>>>>> On 04/20/2013 05:27 PM, Jan Kiszka wrote:
>>>>>>>
>>>>>>>> On 2013-04-20 17:21, Gilles Chanteperdrix wrote:
>>>>>>>>> On 04/20/2013 05:18 PM, Jan Kiszka wrote:
>>>>>>>>>
>>>>>>>>>> On 2013-04-20 17:14, Gilles Chanteperdrix wrote:
>>>>>>>>>>> On 04/20/2013 10:19 AM, Jan Kiszka wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 2013-04-20 08:04, Michael Haberler wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 19.04.2013 um 21:06 schrieb Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 04/19/2013 01:46 PM, Leopold Palomo-Avellaneda wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> http://lists.mech.kuleuven.be/pipermail/orocos-users/2013-April/006986.html
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> that link does not tell us why you need this option. And that would be
>>>>>>>>>>>>>> the most important information.
>>>>>>>>>>>>>
>>>>>>>>>>>>> with the linuxcnc package build I need to turn on --enable-dlopen-skins as well to get Python modules to work properly
>>>>>>>>>>>>
>>>>>>>>>>>> OK, it looks like we should try harder to detect dlopen scenarios during
>>>>>>>>>>>> runtime to avoid build-time switches. This is likely Xenomai 3 material:
>>>>>>>>>>>>
>>>>>>>>>>>>   - We need to disable TLS optimizations by default (no big deal).
>>>>>>>>>>>>
>>>>>>>>>>>>   - In the POSIX skin constructor, we need to read out the mlockall
>>>>>>>>>>>>     state, lock everything if necessary, and restore the state
>>>>>>>>>>>>     accordingly afterward. The Nucleus may help us here if there is no
>>>>>>>>>>>>     adequate libc service (ABI change -> Xenomai 3).
>>>>>>>>>>>>
>>>>>>>>>>>>   - IIRC, the problem with unconditional auto-shadowing back then were
>>>>>>>>>>>>     the improper scheduling parameters that POSIX used to apply. That
>>>>>>>>>>>>     was fixed a while back. So if we simple re-apply the current
>>>>>>>>>>>>     parameters, it should cause no harm in a dlopen scenario. But I need
>>>>>>>>>>>>     to check this again at work against our scenario.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> You do not like the idea of an environment variable allowing to disable
>>>>>>>>>>> the automatic shadowing?
>>>>>>>>>>
>>>>>>>>>> Not as a long-term solution as it is user-unfriendly. But it can be an
>>>>>>>>>> option worth considering for 2.6.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apparently -forge is already doing it. The advantage of this solution is
>>>>>>>>> that the same binary serves well several usages, if we intend to provide
>>>>>>>>> packages as generic as possible, this seems like the way to go. Several
>>>>>>>>> of the changes I made in the last few weeks go in the same direction.
>>>>>>>>
>>>>>>>> I'm not sure if there is added value for the controlling auto-shadowing
>>>>>>>> in general. But for the case it is in conflict with dlopen, the solution
>>>>>>>> I'm proposing is clearly superior as it removes those conflicts
>>>>>>>> automatically.
>>>>>>>>
>>>>>>>> Of course, an environment variable control can exist in parallel if
>>>>>>>> there is a need beyond the dlopen conflict resolution.
>>>>>>>
>>>>>>>
>>>>>>> The difference with what you propose is that you propose a syscall to
>>>>>>> get the mlockall state. Another solution would be not to call munlockall
>>>>>>> after the main thread shadowing, this looks less complicated and does
>>>>>>> not require ABI changes.
>>>>>>
>>>>>> Yes, that's an option as well. But then we should apply this
>>>>>> consistently, invoking mlockall from all skin init functions
>>>>>> unconditionally. The nucleus depends on this anyway. Not sure if such
>>>>>> change would be fine for 2.6 - you decide.
>>>>>
>>>>>
>>>>> What we could do is:
>>>>> - if XENO_NOSHADOW is set, shadow the main thread, and call mlockall
>>>>> - if it is not set, do not shadow the main thread or call
>>>>> mlockall/munlockall
>>>>
>>>> That's not what I suggested. I was questioning the value of _not_ doing
>>>> mlockall automatically during init, thus reducing user duties. That
>>>> would reduce the need to think about XENO_NOSHADOW or not as a normal user.
>>>
>>>
>>> Currently, when the posix skin library start, it does:
>>> mlockall
>>> shadow main thread
>>> munlockall
>>>
>>> Now, the munlockall is certainly an issue when dlopening
>>>
>>> What I propose instead is to do:
>>> if (!getenv("XENO_NOSHADOW")) {
>>> 	mlockall
>>> 	shadow main thread
>>> }
>>>
>>> That will avoid the problem with calling munlockall, and if people who
>>> currently use --enable-dlopen-skins really want to avoid shadowing the
>>> main thread (which I doubt), they can set the environment variable
>>> XENO_NOSHADOW.
>>
>> Totally clear. But why still requiring the application to call mlockall
>> if we do not autoshadow or use a different skin? It's pointless to have
>> this explicit call in the application if we can easily do this from the
>> library.
>
>
> Well, we have --enable-posix-auto-mlockall for that. But you are right,
> we get rid of all this and enable automatic mlockall for all skins. I do
> not see any drawbacks.
>
>

 From the dept of Xenomai archive: the original intent of controlling 
when and how memory locking is done was to address this bug with very 
old kernels:

http://lkml.indiana.edu/hypermail/linux/kernel/0310.0/0649.html

Back then, apps mmapping I/O space would then mlock(MCL_CURRENT), 
mmap(I/O space), then mlock(MCL_FUTURE) to work around this bug manually.

-- 
Philippe.

Re: [Xenomai] Default options in the debian package

[Jan Kiszka]

On 2013-04-23 14:40, Philippe Gerum wrote:
>> Well, we have --enable-posix-auto-mlockall for that. But you are right,
>> we get rid of all this and enable automatic mlockall for all skins. I do
>> not see any drawbacks.
>>
>>
> 
>  From the dept of Xenomai archive: the original intent of controlling 
> when and how memory locking is done was to address this bug with very 
> old kernels:
> 
> http://lkml.indiana.edu/hypermail/linux/kernel/0310.0/0649.html
> 
> Back then, apps mmapping I/O space would then mlock(MCL_CURRENT), 
> mmap(I/O space), then mlock(MCL_FUTURE) to work around this bug manually.
> 

Interesting - but luckily no longer relevant. :)

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Default options in the debian package

[Philippe Gerum]

On 04/23/2013 02:55 PM, Jan Kiszka wrote:
> On 2013-04-23 14:40, Philippe Gerum wrote:
>>> Well, we have --enable-posix-auto-mlockall for that. But you are right,
>>> we get rid of all this and enable automatic mlockall for all skins. I do
>>> not see any drawbacks.
>>>
>>>
>>
>>   From the dept of Xenomai archive: the original intent of controlling
>> when and how memory locking is done was to address this bug with very
>> old kernels:
>>
>> http://lkml.indiana.edu/hypermail/linux/kernel/0310.0/0649.html
>>
>> Back then, apps mmapping I/O space would then mlock(MCL_CURRENT),
>> mmap(I/O space), then mlock(MCL_FUTURE) to work around this bug manually.
>>
>
> Interesting - but luckily no longer relevant. :)
>

Agreed.

-- 
Philippe.