[Xenomai] Task resume problem

[Xenomai] Task resume problem

[Stratiyenko Roman]

Hi.

Can anybody help me with one not easy to debug problem.
Problem happens once per week or two weeks.

I have 2 user-level rt-task.
One task (rttask) is scheduled isochronous using rt_task_wait_period() and
have an max priority: 99
Second task (AFrttask) is started once, then it suspends from itself and
resumes from the first task.

Program works as expected, but sometimes (very very rarely) AFrttask not
resuming.
I connected to the running process using GDB and tracked it. rttask works
as expected, It calls resume function of the second task, but second task
is not resuming.

Here is the dump of /proc/xenomai/sched/stat
CPU  PID    MSW        CSW        XSC        PF    STAT       %CPU  NAME
  0  0      0          2537012    0          0     00018000   89.1  [ROOT/0]
  0  25095  14         18         75         0     000680c0    0.0  main
  0  25097  352        25832      25486      0     00048041    0.0  AFrttask
  0  25098  20         1012279    1043897    0     00048042    0.9  rttask
  0  0      0          43139223   0          0     00000000    7.8
 [IRQ16641: [timer]]

I can see that CSW of rttask is incrementing, but CSW of AFtrtask isn't

<stack trace of AFrttask>
[Switching to thread 8 (Thread 0x7f66123e0700 (LWP 25097))]
#0  0x00007f6611cdd394 in __wrap_kill () from
/usr/xenomai/lib/libcobalt.so.2
(gdb) bt
#0  0x00007f6611cdd394 in __wrap_kill () from
/usr/xenomai/lib/libcobalt.so.2
#1  0x00007f6611ef370c in threadobj_suspend ()
   from /usr/xenomai/lib/libcopperplate.so.0
#2  0x00007f661210167c in rt_task_suspend ()
   from /usr/xenomai/lib/libalchemy.so.0
#3  0x000000000043bb6c in AutomationFlow_c::suspend (this=0xd17420)
    at /Compile/t3/src/Common/AutomationFlow.cpp:30
#4  0x0000000000441f39 in PFlow_c::loop (this=0xd17420)
    at /Compile/t3/src/Common/PFlow.cpp:157
#5  0x000000000043b9f3 in APRun (AF=0xd17420)
    at /Compile/t3/src/Common/AutomationFlow.cpp:14
#6  0x00007f6612100d3a in task_entry () from
/usr/xenomai/lib/libalchemy.so.0
#7  0x00007f6611ef262b in thread_trampoline ()
   from /usr/xenomai/lib/libcopperplate.so.0
#8  0x00007f6611cdd76b in cobalt_thread_trampoline ()
   from /usr/xenomai/lib/libcobalt.so.2
#9  0x00007f6611ab6182 in start_thread (arg=0x7f66123e0700)
    at pthread_create.c:312
#10 0x00007f6610db547d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

I run my code on virtualbox x86-64, [Xenomai] Cobalt v3.0-rc7 (Exact Zero)
Host machine works on Windows 7, and I shutdown host PC using hibernization
several times when my xenomai program was running.

Why my task not resuming from suspend? I will not kill my process if any
idea where to look the state of thread using gdb, proc etc. Please help me
understand why this happends.

Thanks.
Roman Stratiienko

Re: [Xenomai] Task resume problem

[Gilles Chanteperdrix]

On Thu, Jan 28, 2016 at 11:37:14AM +0200, Stratiyenko Roman wrote:
> I run my code on virtualbox x86-64, [Xenomai] Cobalt v3.0-rc7
> (Exact Zero)

Do you have the same problem with the latest version of Xenomai?

-- 
					    Gilles.
https://click-hack.org

[Xenomai] Fwd: Task resume problem

[Stratiyenko Roman]

Ah,
After I will install the latest version, I will need to wait week or two or
more to say If this problem still exists.

If no other ideas-I will do it



2016-01-28 11:43 GMT+02:00 Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org>:

> On Thu, Jan 28, 2016 at 11:37:14AM +0200, Stratiyenko Roman wrote:
> > I run my code on virtualbox x86-64, [Xenomai] Cobalt v3.0-rc7
> > (Exact Zero)
>
> Do you have the same problem with the latest version of Xenomai?
>
> --
>                                             Gilles.
> https://click-hack.org
>

Re: [Xenomai] Task resume problem

[Philippe Gerum]

On 01/28/2016 10:37 AM, Stratiyenko Roman wrote:
> Hi.
> 
> Can anybody help me with one not easy to debug problem.
> Problem happens once per week or two weeks.
> 
> I have 2 user-level rt-task.
> One task (rttask) is scheduled isochronous using rt_task_wait_period() and
> have an max priority: 99
> Second task (AFrttask) is started once, then it suspends from itself and
> resumes from the first task.
> 
> Program works as expected, but sometimes (very very rarely) AFrttask not
> resuming.
> I connected to the running process using GDB and tracked it. rttask works
> as expected, It calls resume function of the second task, but second task
> is not resuming.
> 
> Here is the dump of /proc/xenomai/sched/stat
> CPU  PID    MSW        CSW        XSC        PF    STAT       %CPU  NAME
>   0  0      0          2537012    0          0     00018000   89.1  [ROOT/0]
>   0  25095  14         18         75         0     000680c0    0.0  main
>   0  25097  352        25832      25486      0     00048041    0.0  AFrttask

STAT == 0x48041, bit #0 means (still) forcibly suspended
(include/cobalt/uapi/kernel/thread.h), which may mean your tasks raced:
rttask could have attempted to wake up AFrttask when the latter did not
self-suspend yet. Wake up notifications are not buffered.

Task suspend/resume constructs are inherently unsafe by design, they are
present in the API only for the purpose of easing migration from legacy
RTOS environments, but I would definitely recommend _not_ to use them as
common inter-task synchronization mechanisms.

-- 
Philippe.

Re: [Xenomai] Task resume problem

[Stratiyenko Roman]

resume() calls every rttask cycle, so if first resume() will not resume
AFrttask, the second will.

I think the problem can be only if:
1. Some my program memory leaks destroyed rttask structure, so when I call
resume(), the pointers are invalid.
2. The problem is inside xenomai.

I will try to find out where is it.
Thanks for helping.


2016-01-28 12:02 GMT+02:00 Philippe Gerum <rpm@xenomai.org>:

> On 01/28/2016 10:37 AM, Stratiyenko Roman wrote:
> > Hi.
> >
> > Can anybody help me with one not easy to debug problem.
> > Problem happens once per week or two weeks.
> >
> > I have 2 user-level rt-task.
> > One task (rttask) is scheduled isochronous using rt_task_wait_period()
> and
> > have an max priority: 99
> > Second task (AFrttask) is started once, then it suspends from itself and
> > resumes from the first task.
> >
> > Program works as expected, but sometimes (very very rarely) AFrttask not
> > resuming.
> > I connected to the running process using GDB and tracked it. rttask works
> > as expected, It calls resume function of the second task, but second task
> > is not resuming.
> >
> > Here is the dump of /proc/xenomai/sched/stat
> > CPU  PID    MSW        CSW        XSC        PF    STAT       %CPU  NAME
> >   0  0      0          2537012    0          0     00018000   89.1
> [ROOT/0]
> >   0  25095  14         18         75         0     000680c0    0.0  main
> >   0  25097  352        25832      25486      0     00048041    0.0
> AFrttask
>
> STAT == 0x48041, bit #0 means (still) forcibly suspended
> (include/cobalt/uapi/kernel/thread.h), which may mean your tasks raced:
> rttask could have attempted to wake up AFrttask when the latter did not
> self-suspend yet. Wake up notifications are not buffered.
>
> Task suspend/resume constructs are inherently unsafe by design, they are
> present in the API only for the purpose of easing migration from legacy
> RTOS environments, but I would definitely recommend _not_ to use them as
> common inter-task synchronization mechanisms.
>
> --
> Philippe.
>

Re: [Xenomai] Task resume problem

[Philippe Gerum]

On 01/28/2016 11:21 AM, Stratiyenko Roman wrote:
> resume() calls every rttask cycle, so if first resume() will not resume
> AFrttask, the second will.

Except that although resume notifications are not buffered,
suspend/resume call depth is tracked. Once the nesting count has dropped
to zero, no more resume notification is actually sent.

I'm afraid your your assumption is wrong.

> 
> I think the problem can be only if:
> 1. Some my program memory leaks destroyed rttask structure, so when I
> call resume(), the pointers are invalid.
> 2. The problem is inside xenomai.
> 
> I will try to find out where is it.
> Thanks for helping.
> 
> 
> 2016-01-28 12:02 GMT+02:00 Philippe Gerum <rpm@xenomai.org
> <mailto:rpm@xenomai.org>>:
> 
>     On 01/28/2016 10:37 AM, Stratiyenko Roman wrote:
>     > Hi.
>     >
>     > Can anybody help me with one not easy to debug problem.
>     > Problem happens once per week or two weeks.
>     >
>     > I have 2 user-level rt-task.
>     > One task (rttask) is scheduled isochronous using rt_task_wait_period() and
>     > have an max priority: 99
>     > Second task (AFrttask) is started once, then it suspends from itself and
>     > resumes from the first task.
>     >
>     > Program works as expected, but sometimes (very very rarely) AFrttask not
>     > resuming.
>     > I connected to the running process using GDB and tracked it. rttask works
>     > as expected, It calls resume function of the second task, but second task
>     > is not resuming.
>     >
>     > Here is the dump of /proc/xenomai/sched/stat
>     > CPU  PID    MSW        CSW        XSC        PF    STAT       %CPU  NAME
>     >   0  0      0          2537012    0          0     00018000   89.1  [ROOT/0]
>     >   0  25095  14         18         75         0     000680c0    0.0  main
>     >   0  25097  352        25832      25486      0     00048041    0.0  AFrttask
> 
>     STAT == 0x48041, bit #0 means (still) forcibly suspended
>     (include/cobalt/uapi/kernel/thread.h), which may mean your tasks raced:
>     rttask could have attempted to wake up AFrttask when the latter did not
>     self-suspend yet. Wake up notifications are not buffered.
> 
>     Task suspend/resume constructs are inherently unsafe by design, they are
>     present in the API only for the purpose of easing migration from legacy
>     RTOS environments, but I would definitely recommend _not_ to use them as
>     common inter-task synchronization mechanisms.
> 
>     --
>     Philippe.
> 
> 


-- 
Philippe.

Re: [Xenomai] Task resume problem

[Stratiyenko Roman]

Thank you for your help

Suspend/resume was replaced by semaphore pend/signal
Now I test the program.

2016-01-28 12:40 GMT+02:00 Philippe Gerum <rpm@xenomai.org>:

> On 01/28/2016 11:21 AM, Stratiyenko Roman wrote:
> > resume() calls every rttask cycle, so if first resume() will not resume
> > AFrttask, the second will.
>
> Except that although resume notifications are not buffered,
> suspend/resume call depth is tracked. Once the nesting count has dropped
> to zero, no more resume notification is actually sent.
>
> I'm afraid your your assumption is wrong.
>
> >
> > I think the problem can be only if:
> > 1. Some my program memory leaks destroyed rttask structure, so when I
> > call resume(), the pointers are invalid.
> > 2. The problem is inside xenomai.
> >
> > I will try to find out where is it.
> > Thanks for helping.
> >
> >
> > 2016-01-28 12:02 GMT+02:00 Philippe Gerum <rpm@xenomai.org
> > <mailto:rpm@xenomai.org>>:
> >
> >     On 01/28/2016 10:37 AM, Stratiyenko Roman wrote:
> >     > Hi.
> >     >
> >     > Can anybody help me with one not easy to debug problem.
> >     > Problem happens once per week or two weeks.
> >     >
> >     > I have 2 user-level rt-task.
> >     > One task (rttask) is scheduled isochronous using
> rt_task_wait_period() and
> >     > have an max priority: 99
> >     > Second task (AFrttask) is started once, then it suspends from
> itself and
> >     > resumes from the first task.
> >     >
> >     > Program works as expected, but sometimes (very very rarely)
> AFrttask not
> >     > resuming.
> >     > I connected to the running process using GDB and tracked it.
> rttask works
> >     > as expected, It calls resume function of the second task, but
> second task
> >     > is not resuming.
> >     >
> >     > Here is the dump of /proc/xenomai/sched/stat
> >     > CPU  PID    MSW        CSW        XSC        PF    STAT
>  %CPU  NAME
> >     >   0  0      0          2537012    0          0     00018000
>  89.1  [ROOT/0]
> >     >   0  25095  14         18         75         0     000680c0
> 0.0  main
> >     >   0  25097  352        25832      25486      0     00048041
> 0.0  AFrttask
> >
> >     STAT == 0x48041, bit #0 means (still) forcibly suspended
> >     (include/cobalt/uapi/kernel/thread.h), which may mean your tasks
> raced:
> >     rttask could have attempted to wake up AFrttask when the latter did
> not
> >     self-suspend yet. Wake up notifications are not buffered.
> >
> >     Task suspend/resume constructs are inherently unsafe by design, they
> are
> >     present in the API only for the purpose of easing migration from
> legacy
> >     RTOS environments, but I would definitely recommend _not_ to use
> them as
> >     common inter-task synchronization mechanisms.
> >
> >     --
> >     Philippe.
> >
> >
>
>
> --
> Philippe.
>