[Xenomai-help] App with pthreads using rt_task_shadow locks up

[Xenomai-help] App with pthreads using rt_task_shadow locks up

[Jochen Behnke]

Hello,

I am currently porting software form a windows based control system to Xenomai. The application creates 3 realtime tasks which build the core of the control unit. Up to this point everything worked fine. The next milestone of this project is to integrate network communication using sockets and pthreads with the control system. The implementation of my - let's call it "SimpleRPC" -  library uses sockets and pthreads. At runtime this library creates at least 3 pthreads (1 that listens for incoming connections, 1 that performs socket I/O per connected client, 1 that executes the requested functions). As the functions, that are called by the workerthread, use xenomai mutexes to synchronize critical sections of code, the workerthread needs to be a realtime task scheduled by xenomai. So I use rt_task_shadow, to "convert" the pthread into a xenomai realtime task. When this code is executed, my application locks up and I have to restart the whole system. When I remove the rt_task_shadow call everything is fine, except for the calls to rt_mutex_lock that return an error code because the calling context is not a realtime task.

I am using
Xenomai 2.2.3
Linux Kernel 2.6.17

What's wrong ?
How can I debug this problem ?

Any hint is appreciated.


Jochen


P.S.:
Is there an easy way to search the xenomai mail archives ? At the moment I walk through them more or less manually.




_________________________________________________________________________
Mit der Gruppen-SMS von WEB.DE FreeMail können Sie eine SMS an alle 
Freunde gleichzeitig schicken: http://freemail.web.de/features/?mc=021179

Re: [Xenomai-help] App with pthreads using rt_task_shadow locks up

[Gilles Chanteperdrix]

Jochen Behnke wrote:
> Hello,
> 
> I am currently porting software form a windows based control system to Xenomai. The application creates 3 realtime tasks which build the core of the control unit. Up to this point everything worked fine. The next milestone of this project is to integrate network communication using sockets and pthreads with the control system. The implementation of my - let's call it "SimpleRPC" -  library uses sockets and pthreads. At runtime this library creates at least 3 pthreads (1 that listens for incoming connections, 1 that performs socket I/O per connected client, 1 that executes the requested functions). As the functions, that are called by the workerthread, use xenomai mutexes to synchronize critical sections of code, the workerthread needs to be a realtime task scheduled by xenomai. So I use rt_task_shadow, to "convert" the pthread into a xenomai realtime task. When this code is executed, my application locks up and I have to restart the whole system. When I remove the rt_task_s
hadow call everything is fine, except for the calls to rt_mutex_lock that return an error code because the calling context is not a realtime task.
> 
> I am using
> Xenomai 2.2.3
> Linux Kernel 2.6.17
> 
> What's wrong ?
> How can I debug this problem ?

Please try your example on the latest release of Xenomai. If it still
fails post here an example which fails, reduced to the minimum.

> 
> Any hint is appreciated.
> 
> 
> Jochen
> 
> 
> P.S.:
> Is there an easy way to search the xenomai mail archives ? At the moment I walk through them more or less manually.

Use google, simply add site:mail.gna.org at the end.

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] App with pthreads using rt_task_shadow locks up

[Gilles Chanteperdrix]

Jochen Behnke wrote:
> Hello Gilles,
> 
> thanks for your response. 
> At the moment I try to cut down my application to the minimum.
> I also will install the latest version of the xenomai 2.2.x branch (2.2.5) and check whether I can reproduce the fault.  As I noticed that some functions have been changed (rt_mutex_lock/unlock no longer exist in Xenomai 2.3.x, they obviously have been renamed for some reason) I would rather stick to the 2.2.x branch. Is that reasonable in your opinion ?
> Will the 2.2.x branch be maintained in the future ?
> 
> Thanks again.
> 
> Jochen

Please do not forget to CC the list.

rt_mutex_lock/rt_mutex_unlock were renamed
rt_mutex_acquire/rt_mutex_release to avoid a conflict, as documented in
API.CHANGES

Please try with Xenomai 2.3.0, or better with trunk. The aim of this
check is for us to avoid chasing bugs that were already solved. For
example if you are using fork() in your application, only I-pipe patches
in the trunk will solve your problem.

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] App with pthreads using rt_task_shadow locks up

[Jochen Behnke]

Hello Gilles,

> Jochen Behnke wrote:
> > Hello Gilles,
> > 
> > thanks for your response. 
> > At the moment I try to cut down my application to the minimum.
> > I also will install the latest version of the xenomai 2.2.x branch (2.2.5) and check whether I can reproduce the fault.  As I noticed that some functions have been changed (rt_mutex_lock/unlock no longer exist in Xenomai 2.3.x, they obviously have been renamed for some reason) I would rather stick to the 2.2.x branch. Is that reasonable in your opinion ?
> > Will the 2.2.x branch be maintained in the future ?
> > 
> > Thanks again.
> > 
> > Jochen
> 
> Please do not forget to CC the list.
I'm sorry. I just used the wrong reply button.

> 
> rt_mutex_lock/rt_mutex_unlock were renamed
> rt_mutex_acquire/rt_mutex_release to avoid a conflict, as documented in
> API.CHANGES
I will have a look at this.

> 
> Please try with Xenomai 2.3.0, or better with trunk. The aim of this
> check is for us to avoid chasing bugs that were already solved. For
> example if you are using fork() in your application, only I-pipe patches
> in the trunk will solve your problem.
At the moment I don't use fork() but I can try using 2.3.x.
I hope that I don't have to change too much code on my side.

As soon as I have results, I'll post them here.

Jochen

_______________________________________________________________________
Viren-Scan für Ihren PC! Jetzt für jeden. Sofort, online und kostenlos.
Gleich testen! http://www.pc-sicherheit.web.de/freescan/?mc=022222

Re: [Xenomai-help] App with pthreads using rt_task_shadow locks up

[Jochen Behnke]

Hello,

after hours of searching, I have finally found the problem. 
It was a bug in my "SimpleRPC" library. The thread function, responsible for the "lockup", mainly constists of an endless loop ( while(pObj->nRun) ) that polls a request queue. This was not a problem as long as this thread was a standard pthread. Turning this thread into a real-time task by adding a call to rt_task_shadow at the beginning of the thread function (before the while loop) resulted in a frozen linux system.

I used to work with the following system
Gentoo Linux
Linux Kernel 2.6.17 (IPIPE 1.4)
Xenomai 2.2.3

Now that I know, that upgrading to Xenomai 2.3.0 is not a problem, I plan to upgrade to the 2.3.x branch.


Regards

Jochen


> 
> Hello Gilles,
> 
> > Jochen Behnke wrote:
> > > Hello Gilles,
> > > 
> > > thanks for your response. 
> > > At the moment I try to cut down my application to the minimum.
> > > I also will install the latest version of the xenomai 2.2.x branch (2.2.5) and check whether I can reproduce the fault.  As I noticed that some functions have been changed (rt_mutex_lock/unlock no longer exist in Xenomai 2.3.x, they obviously have been renamed for some reason) I would rather stick to the 2.2.x branch. Is that reasonable in your opinion ?
> > > Will the 2.2.x branch be maintained in the future ?
> > > 
> > > Thanks again.
> > > 
> > > Jochen
> > 
> > Please do not forget to CC the list.
> I'm sorry. I just used the wrong reply button.
> 
> > 
> > rt_mutex_lock/rt_mutex_unlock were renamed
> > rt_mutex_acquire/rt_mutex_release to avoid a conflict, as documented in
> > API.CHANGES
> I will have a look at this.
> 
> > 
> > Please try with Xenomai 2.3.0, or better with trunk. The aim of this
> > check is for us to avoid chasing bugs that were already solved. For
> > example if you are using fork() in your application, only I-pipe patches
> > in the trunk will solve your problem.
> At the moment I don't use fork() but I can try using 2.3.x.
> I hope that I don't have to change too much code on my side.
> 
> As soon as I have results, I'll post them here.
> 
> Jochen
> 
> _______________________________________________________________________
> Viren-Scan für Ihren PC! Jetzt für jeden. Sofort, online und kostenlos.
> Gleich testen! http://www.pc-sicherheit.web.de/freescan/?mc=022222
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
> 


_____________________________________________________________________
Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
http://smartsurfer.web.de/?mc=100071&distributionid=000000000066

Re: [Xenomai-help] App with pthreads using rt_task_shadow locks up

[Philippe Gerum]

On Thu, 2007-03-15 at 17:17 +0100, Jochen Behnke wrote:
> Hello,
> 
> after hours of searching, I have finally found the problem. 
> It was a bug in my "SimpleRPC" library. The thread function, responsible for the "lockup", mainly constists of an endless loop ( while(pObj->nRun) ) that polls a request queue. This was not a problem as long as this thread was a standard pthread. Turning this thread into a real-time task by adding a call to rt_task_shadow at the beginning of the thread function (before the while loop) resulted in a frozen linux system.
> 
> I used to work with the following system
> Gentoo Linux
> Linux Kernel 2.6.17 (IPIPE 1.4)
> Xenomai 2.2.3
> 
> Now that I know, that upgrading to Xenomai 2.3.0 is not a problem, I plan to upgrade to the 2.3.x branch.
> 

You may detect such kind of issues switching on the
CONFIG_XENO_OPT_WATCHDOG option in the kernel configuration. Normally,
this would pull the brake after more than 4 seconds of uninterrupted
real-time activity in primary mode (i.e. without yielding control to the
regular Linux activities).

> 
> Regards
> 
> Jochen
> 
> 
> > 
> > Hello Gilles,
> > 
> > > Jochen Behnke wrote:
> > > > Hello Gilles,
> > > > 
> > > > thanks for your response. 
> > > > At the moment I try to cut down my application to the minimum.
> > > > I also will install the latest version of the xenomai 2.2.x branch (2.2.5) and check whether I can reproduce the fault.  As I noticed that some functions have been changed (rt_mutex_lock/unlock no longer exist in Xenomai 2.3.x, they obviously have been renamed for some reason) I would rather stick to the 2.2.x branch. Is that reasonable in your opinion ?
> > > > Will the 2.2.x branch be maintained in the future ?
> > > > 
> > > > Thanks again.
> > > > 
> > > > Jochen
> > > 
> > > Please do not forget to CC the list.
> > I'm sorry. I just used the wrong reply button.
> > 
> > > 
> > > rt_mutex_lock/rt_mutex_unlock were renamed
> > > rt_mutex_acquire/rt_mutex_release to avoid a conflict, as documented in
> > > API.CHANGES
> > I will have a look at this.
> > 
> > > 
> > > Please try with Xenomai 2.3.0, or better with trunk. The aim of this
> > > check is for us to avoid chasing bugs that were already solved. For
> > > example if you are using fork() in your application, only I-pipe patches
> > > in the trunk will solve your problem.
> > At the moment I don't use fork() but I can try using 2.3.x.
> > I hope that I don't have to change too much code on my side.
> > 
> > As soon as I have results, I'll post them here.
> > 
> > Jochen
> > 
> > _______________________________________________________________________
> > Viren-Scan für Ihren PC! Jetzt für jeden. Sofort, online und kostenlos.
> > Gleich testen! http://www.pc-sicherheit.web.de/freescan/?mc=022222
> > 
> > 
> > _______________________________________________
> > Xenomai-help mailing list
> > Xenomai-help@domain.hid
> > https://mail.gna.org/listinfo/xenomai-help
> > 
> 
> 
> _____________________________________________________________________
> Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
> http://smartsurfer.web.de/?mc=100071&distributionid=000000000066
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] App with pthreads using rt_task_shadow locks up

[Jochen Behnke]

Hello Philippe,

thanks for your response.

> On Thu, 2007-03-15 at 17:17 +0100, Jochen Behnke wrote:
> > Hello,
> > 
> > after hours of searching, I have finally found the problem. 
> > It was a bug in my "SimpleRPC" library. The thread function, responsible for the "lockup", mainly constists of an endless loop ( while(pObj->nRun) ) that polls a request queue. This was not a problem as long as this thread was a standard pthread. Turning this thread into a real-time task by adding a call to rt_task_shadow at the beginning of the thread function (before the while loop) resulted in a frozen linux system.
> > 
> > I used to work with the following system
> > Gentoo Linux
> > Linux Kernel 2.6.17 (IPIPE 1.4)
> > Xenomai 2.2.3
> > 
> > Now that I know, that upgrading to Xenomai 2.3.0 is not a problem, I plan to upgrade to the 2.3.x branch.
> > 
> 
> You may detect such kind of issues switching on the
> CONFIG_XENO_OPT_WATCHDOG option in the kernel configuration. Normally,
> this would pull the brake after more than 4 seconds of uninterrupted
> real-time activity in primary mode (i.e. without yielding control to the
> regular Linux activities).

That's great. I will check this out.

BTW, is there an easy way, other than reboot,  to release xenomai ressources (mutexes,...) that have not been freed after an application crash ? 

Thanks in advance.

Jochen


> 
> > 
> > Regards
> > 
> > Jochen
> > 
> > 
> > > 
> > > Hello Gilles,
> > > 
> > > > Jochen Behnke wrote:
> > > > > Hello Gilles,
> > > > > 
> > > > > thanks for your response. 
> > > > > At the moment I try to cut down my application to the minimum.
> > > > > I also will install the latest version of the xenomai 2.2.x branch (2.2.5) and check whether I can reproduce the fault.  As I noticed that some functions have been changed (rt_mutex_lock/unlock no longer exist in Xenomai 2.3.x, they obviously have been renamed for some reason) I would rather stick to the 2.2.x branch. Is that reasonable in your opinion ?
> > > > > Will the 2.2.x branch be maintained in the future ?
> > > > > 
> > > > > Thanks again.
> > > > > 
> > > > > Jochen
> > > > 
> > > > Please do not forget to CC the list.
> > > I'm sorry. I just used the wrong reply button.
> > > 
> > > > 
> > > > rt_mutex_lock/rt_mutex_unlock were renamed
> > > > rt_mutex_acquire/rt_mutex_release to avoid a conflict, as documented in
> > > > API.CHANGES
> > > I will have a look at this.
> > > 
> > > > 
> > > > Please try with Xenomai 2.3.0, or better with trunk. The aim of this
> > > > check is for us to avoid chasing bugs that were already solved. For
> > > > example if you are using fork() in your application, only I-pipe patches
> > > > in the trunk will solve your problem.
> > > At the moment I don't use fork() but I can try using 2.3.x.
> > > I hope that I don't have to change too much code on my side.
> > > 
> > > As soon as I have results, I'll post them here.
> > > 
> > > Jochen
> > > 
> > > 
> > > _______________________________________________
> > > Xenomai-help mailing list
> > > Xenomai-help@domain.hid
> > > https://mail.gna.org/listinfo/xenomai-help
> > > 
> > 
> > _______________________________________________
> > Xenomai-help mailing list
> > Xenomai-help@domain.hid
> > https://mail.gna.org/listinfo/xenomai-help
> -- 
> Philippe.

_____________________________________________________________________
Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
http://smartsurfer.web.de/?mc=100071&distributionid=000000000066

Re: [Xenomai-help] App with pthreads using rt_task_shadow locks up

[Philippe Gerum]

On Thu, 2007-03-15 at 19:34 +0100, Jochen Behnke wrote:
> Hello Philippe,
> 
> thanks for your response.
> 
> > On Thu, 2007-03-15 at 17:17 +0100, Jochen Behnke wrote:
> > > Hello,
> > > 
> > > after hours of searching, I have finally found the problem. 
> > > It was a bug in my "SimpleRPC" library. The thread function, responsible for the "lockup", mainly constists of an endless loop ( while(pObj->nRun) ) that polls a request queue. This was not a problem as long as this thread was a standard pthread. Turning this thread into a real-time task by adding a call to rt_task_shadow at the beginning of the thread function (before the while loop) resulted in a frozen linux system.
> > > 
> > > I used to work with the following system
> > > Gentoo Linux
> > > Linux Kernel 2.6.17 (IPIPE 1.4)
> > > Xenomai 2.2.3
> > > 
> > > Now that I know, that upgrading to Xenomai 2.3.0 is not a problem, I plan to upgrade to the 2.3.x branch.
> > > 
> > 
> > You may detect such kind of issues switching on the
> > CONFIG_XENO_OPT_WATCHDOG option in the kernel configuration. Normally,
> > this would pull the brake after more than 4 seconds of uninterrupted
> > real-time activity in primary mode (i.e. without yielding control to the
> > regular Linux activities).
> 
> That's great. I will check this out.
> 
> BTW, is there an easy way, other than reboot,  to release xenomai ressources (mutexes,...) that have not been freed after an application crash ? 
> 

This depends on the skin being used. The native one does not free the
resources upon application exit yet (this feature is on the todo list
for the upcoming 2.4 though), OTOH the POSIX skin does. A work-around
with the native skin is to use constructs like the following one in your
app to recycle the left overs; not pretty, but works, until we provide
an automated support:

if (rt_mutex_bind("some_mutex", &mutex_desc, ...) == 0) {
	rt_mutex_delete(&mutex_desc);
}
err = rt_mutex_create("some_mutex", &mutex_desc);

-- 
Philippe.

Re: [Xenomai-help] App with pthreads using rt_task_shadow locks up

[Jochen Behnke]

Philippe,

the workaround for the native skin is OK for me.
Thanks for your help :-)

Regards

Jochen


> On Thu, 2007-03-15 at 19:34 +0100, Jochen Behnke wrote:
> > Hello Philippe,
> > 
> > thanks for your response.
> > 
> > > On Thu, 2007-03-15 at 17:17 +0100, Jochen Behnke wrote:
> > > > Hello,
> > > > 
> > > > after hours of searching, I have finally found the problem. 
> > > > It was a bug in my "SimpleRPC" library. The thread function, responsible for the "lockup", mainly constists of an endless loop ( while(pObj->nRun) ) that polls a request queue. This was not a problem as long as this thread was a standard pthread. Turning this thread into a real-time task by adding a call to rt_task_shadow at the beginning of the thread function (before the while loop) resulted in a frozen linux system.
> > > > 
> > > > I used to work with the following system
> > > > Gentoo Linux
> > > > Linux Kernel 2.6.17 (IPIPE 1.4)
> > > > Xenomai 2.2.3
> > > > 
> > > > Now that I know, that upgrading to Xenomai 2.3.0 is not a problem, I plan to upgrade to the 2.3.x branch.
> > > > 
> > > 
> > > You may detect such kind of issues switching on the
> > > CONFIG_XENO_OPT_WATCHDOG option in the kernel configuration. Normally,
> > > this would pull the brake after more than 4 seconds of uninterrupted
> > > real-time activity in primary mode (i.e. without yielding control to the
> > > regular Linux activities).
> > 
> > That's great. I will check this out.
> > 
> > BTW, is there an easy way, other than reboot,  to release xenomai ressources (mutexes,...) that have not been freed after an application crash ? 
> > 
> 
> This depends on the skin being used. The native one does not free the
> resources upon application exit yet (this feature is on the todo list
> for the upcoming 2.4 though), OTOH the POSIX skin does. A work-around
> with the native skin is to use constructs like the following one in your
> app to recycle the left overs; not pretty, but works, until we provide
> an automated support:
> 
> if (rt_mutex_bind("some_mutex", &mutex_desc, ...) == 0) {
> 	rt_mutex_delete(&mutex_desc);
> }
> err = rt_mutex_create("some_mutex", &mutex_desc);
> 
> -- 
> Philippe.
> 
> 
> 


_______________________________________________________________________
Viren-Scan für Ihren PC! Jetzt für jeden. Sofort, online und kostenlos.
Gleich testen! http://www.pc-sicherheit.web.de/freescan/?mc=022222