[Xenomai-help] how to stop a runaway task

[Xenomai-help] how to stop a runaway task

[Carsten Spieß]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

Hello,

i'm running Xenomai 2.3.2 on a PPC target with Linux 2.4.25 kernel.
I trying to programm a supervising task, that detects a runaway task
and suspend it.

My aproach was to make a high priority periodic task to look if
the low prio task is running correctly and suspend it when it detects
failure.
But this approach didn't work, the higer prio task isn't called unless
the lower prio task gives up cpu (by calling e.g. rt_task_yield or
nanosleep) (see attached demo source)

When not being cooperative (removing the rt_task_yield in the demo)
my system is reset after some seconds.

Why isn't my supervising task scheduled?

Tanks for help, regards

Carsten
-- 
Dipl.-Ing. Carsten Spieß
Softwareentwicklung - Beratung - Schulung
Lauterlech 39d
D-86152 Augsburg
Tel.: +49(821)15 999 765
Fax.: +49(821)15 999 767
mailto://mail@domain.hid
http://www.carsten-spiess.de

[-- Attachment #2: demo.c --]
[-- Type: application/octet-stream, Size: 2733 bytes --]

#include <stdio.h>
#include <native/task.h>
#include <signal.h>
#include <sys/mman.h>

static int count;

static RT_TASK id_low;
static RT_TASK id_high;

#define LOW_PRIO 5
#define HIGH_PRIO 10

void init_task(int n);

void runaway_task(void* dummy)
{
   int i;
   struct timespec tstart;
   struct timespec tperiod;
   
   tstart.tv_sec = 0;
   tstart.tv_nsec = 0;
   tperiod.tv_sec = 0;
   tperiod.tv_nsec = 16000;  // zeit in ns, reset bei <16 µs 

   printf("runaway task\n");

   while (1)
   {
//      sleep(1);
//      nanosleep(&tperiod,&tstart);
      rt_task_yield();
      for(i=0;i<50000;i++)count++;
   }
}

void supervisor_task(void* dummy)
{
   unsigned long nOverrun;
   RTIME   tick, tick_old;
   
   int step = 0;
   int loop = 0;

   if (rt_task_set_periodic(&id_high,TM_NOW, rt_timer_ns2ticks(500000)) == 0)
   {
      printf("supervisor task\n");

      tick_old = rt_timer_tsc();
      while (1)
      {
         //nanosleep(&tperiod,&tstart);
         if (rt_task_wait_period(&nOverrun) != 0)
            printf("supervisor task overrun %lu\n", nOverrun);

         tick = rt_timer_tsc();
         
         loop ++;
         switch (step)
         {
         case 0:
            if(loop > 100)
            {
               loop = 0;
               step = 1;
               init_task(LOW_PRIO);
            }
            break;
         case 1:
            if(loop > 200)
            {
               loop = 0;
               step = 2;
               rt_task_suspend(&id_low);
               printf("count = %d %lu %llu\n", count,nOverrun,tick-tick_old);
            }
            break;
         case 2:
            if(loop > 20)
            {
               loop = 0;
               step = 1;
               rt_task_resume(&id_low);
            }
            break;
         }
         tick_old = tick;
      }
   }
}

void init_task(int prio)
{
   switch (prio)
   {
   case LOW_PRIO:
      rt_task_create(&id_low, "runaway", 8192,prio, 0);
      rt_task_start(&id_low,runaway_task,0);
      break;
   case HIGH_PRIO:
      rt_task_create(&id_high, "supervisor", 8192,prio, 0);
      rt_task_start(&id_high,supervisor_task,0);
      break;
   }
}

void init_signals(sigset_t *mask)
{
   sigemptyset(mask);
   sigaddset(mask,SIGINT);
   sigaddset(mask,SIGTERM);
   sigaddset(mask,SIGHUP);
   sigaddset(mask,SIGALRM);

   pthread_sigmask(SIG_BLOCK, mask, NULL);
}

int main(int ac, char *av[])
{
   sigset_t mask;
   int sig;

   mlockall(MCL_CURRENT|MCL_FUTURE);
   count = 0;
   
   init_signals(&mask);
   init_task(HIGH_PRIO);
   
   sigwait(&mask, &sig);

   return 0;
}

Re: [Xenomai-help] how to stop a runaway task

[Philippe Gerum]

Carsten Spieß wrote:
> Hello,
> 
> i'm running Xenomai 2.3.2 on a PPC target with Linux 2.4.25 kernel.
> I trying to programm a supervising task, that detects a runaway task
> and suspend it.
> 
> My aproach was to make a high priority periodic task to look if
> the low prio task is running correctly and suspend it when it detects
> failure.
> But this approach didn't work, the higer prio task isn't called unless
> the lower prio task gives up cpu (by calling e.g. rt_task_yield or
> nanosleep) (see attached demo source)

This would basically say that the core does not give any real-time
guarantee. Please send some demo code showing this.

> 
> When not being cooperative (removing the rt_task_yield in the demo)
> my system is reset after some seconds.
> 
> Why isn't my supervising task scheduled?
> 

Try enabling the nucleus watchdog (Xenomai debug options). It does kill
runaway tasks. If this works on your setup, then your code must be
wrong. If it does not trigger, then we need to understand why  -- there
is one case where it may not trigger, but this does not involve any
breakage in task management.

> Tanks for help, regards
> 
> Carsten
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help


-- 
Philippe.

Re: [Xenomai-help] how to stop a runaway task

[Carsten Spieß]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

Hello Philippe,

>> But this approach didn't work, the higer prio task isn't called unless
>> the lower prio task gives up cpu (by calling e.g. rt_task_yield or
>> nanosleep) (see attached demo source)
> This would basically say that the core does not give any real-time
> guarantee.
I hope this is not the case.
> Please send some demo code showing this.
I've attached it, please remove the rt_task_yield to see the effect.
>> 
>> When not being cooperative (removing the rt_task_yield in the demo)
>> my system is reset after some seconds.
>> 
>> Why isn't my supervising task scheduled?
>> 

> Try enabling the nucleus watchdog (Xenomai debug options). It does kill
> runaway tasks.
O.k. i will try this.

> If this works on your setup, then your code must be
> wrong. If it does not trigger, then we need to understand why  -- there
> is one case where it may not trigger, but this does not involve any
> breakage in task management.

Regards Carsten

[-- Attachment #2: demo.c --]
[-- Type: application/octet-stream, Size: 2733 bytes --]

#include <stdio.h>
#include <native/task.h>
#include <signal.h>
#include <sys/mman.h>

static int count;

static RT_TASK id_low;
static RT_TASK id_high;

#define LOW_PRIO 5
#define HIGH_PRIO 10

void init_task(int n);

void runaway_task(void* dummy)
{
   int i;
   struct timespec tstart;
   struct timespec tperiod;
   
   tstart.tv_sec = 0;
   tstart.tv_nsec = 0;
   tperiod.tv_sec = 0;
   tperiod.tv_nsec = 16000;  // zeit in ns, reset bei <16 µs 

   printf("runaway task\n");

   while (1)
   {
//      sleep(1);
//      nanosleep(&tperiod,&tstart);
      rt_task_yield();
      for(i=0;i<50000;i++)count++;
   }
}

void supervisor_task(void* dummy)
{
   unsigned long nOverrun;
   RTIME   tick, tick_old;
   
   int step = 0;
   int loop = 0;

   if (rt_task_set_periodic(&id_high,TM_NOW, rt_timer_ns2ticks(500000)) == 0)
   {
      printf("supervisor task\n");

      tick_old = rt_timer_tsc();
      while (1)
      {
         //nanosleep(&tperiod,&tstart);
         if (rt_task_wait_period(&nOverrun) != 0)
            printf("supervisor task overrun %lu\n", nOverrun);

         tick = rt_timer_tsc();
         
         loop ++;
         switch (step)
         {
         case 0:
            if(loop > 100)
            {
               loop = 0;
               step = 1;
               init_task(LOW_PRIO);
            }
            break;
         case 1:
            if(loop > 200)
            {
               loop = 0;
               step = 2;
               rt_task_suspend(&id_low);
               printf("count = %d %lu %llu\n", count,nOverrun,tick-tick_old);
            }
            break;
         case 2:
            if(loop > 20)
            {
               loop = 0;
               step = 1;
               rt_task_resume(&id_low);
            }
            break;
         }
         tick_old = tick;
      }
   }
}

void init_task(int prio)
{
   switch (prio)
   {
   case LOW_PRIO:
      rt_task_create(&id_low, "runaway", 8192,prio, 0);
      rt_task_start(&id_low,runaway_task,0);
      break;
   case HIGH_PRIO:
      rt_task_create(&id_high, "supervisor", 8192,prio, 0);
      rt_task_start(&id_high,supervisor_task,0);
      break;
   }
}

void init_signals(sigset_t *mask)
{
   sigemptyset(mask);
   sigaddset(mask,SIGINT);
   sigaddset(mask,SIGTERM);
   sigaddset(mask,SIGHUP);
   sigaddset(mask,SIGALRM);

   pthread_sigmask(SIG_BLOCK, mask, NULL);
}

int main(int ac, char *av[])
{
   sigset_t mask;
   int sig;

   mlockall(MCL_CURRENT|MCL_FUTURE);
   count = 0;
   
   init_signals(&mask);
   init_task(HIGH_PRIO);
   
   sigwait(&mask, &sig);

   return 0;
}

Re: [Xenomai-help] how to stop a runaway task

[Philippe Gerum]

Carsten Spieß wrote:
> Hello Philippe,
> 
>>> But this approach didn't work, the higer prio task isn't called unless
>>> the lower prio task gives up cpu (by calling e.g. rt_task_yield or
>>> nanosleep) (see attached demo source)
>> This would basically say that the core does not give any real-time
>> guarantee.
> I hope this is not the case.
>> Please send some demo code showing this.
> I've attached it, please remove the rt_task_yield to see the effect.
>>> When not being cooperative (removing the rt_task_yield in the demo)
>>> my system is reset after some seconds.

This works fine on 2.4-rc4/x86-2.6.22 here, so I need to switch to a
similar ppc configuration to try reproducing it. More news later. Please
send your kernel .config for 2.4.25 in the meantime.

>>>
>>> Why isn't my supervising task scheduled?
>>>
> 
>> Try enabling the nucleus watchdog (Xenomai debug options). It does kill
>> runaway tasks.
> O.k. i will try this.

You will have to remove the printf() from the low prio task for the
nucleus watchdog to detect the runaway situation, otherwise, your task
would be considered as being under the control of the Linux scheduler
until it issues a blocking Xenomai syscall (which it never does), and
therefore escape the watchdog detection. That's the particular case I
was talking about.

> 
>> If this works on your setup, then your code must be
>> wrong. If it does not trigger, then we need to understand why  -- there
>> is one case where it may not trigger, but this does not involve any
>> breakage in task management.
> 
> Regards Carsten


-- 
Philippe.

Re: [Xenomai-help] how to stop a runaway task

[Carsten Spieß]

Hello Philippe,


> You will have to remove the printf() from the low prio task for the
> nucleus watchdog to detect the runaway situation, otherwise, your task
> would be considered as being under the control of the Linux scheduler
> until it issues a blocking Xenomai syscall (which it never does), and
> therefore escape the watchdog detection. That's the particular case I
> was talking about.
Meanwhile i got my sample code running:
When removing the printf it works fine.
So for me it looks like a problem when a task is switched to secondary
domain.
Is there a way to force a thread to switch back to primary domain?

Now i unsuccesfully tried to get a posix skin variant running.
(my real app is based on pthread)
Since i only can use signals to suspend a posix thread (correct me if
i'm wrong) i run in the same problem with the secondary domain.

Is there a way to switch back to primary domain in posix skin?

Do i have a chance to do this with pthread or do i have to rewrite my
code using rt_task* calls?

Thanks for your help, regards

Carsten
-- 
Dipl.-Ing. Carsten Spieß
Softwareentwicklung - Beratung - Schulung
Lauterlech 39d
D-86152 Augsburg
Tel.: +49(821)15 999 765
Fax.: +49(821)15 999 767
mailto://mail@domain.hid
http://www.carsten-spiess.de

Re: [Xenomai-help] how to stop a runaway task

[Gilles Chanteperdrix]

Carsten Spieß wrote:
 > Hello Philippe,
 > 
 > 
 > > You will have to remove the printf() from the low prio task for the
 > > nucleus watchdog to detect the runaway situation, otherwise, your task
 > > would be considered as being under the control of the Linux scheduler
 > > until it issues a blocking Xenomai syscall (which it never does), and
 > > therefore escape the watchdog detection. That's the particular case I
 > > was talking about.
 > Meanwhile i got my sample code running:
 > When removing the printf it works fine.
 > So for me it looks like a problem when a task is switched to secondary
 > domain.
 > Is there a way to force a thread to switch back to primary domain?
 > 
 > Now i unsuccesfully tried to get a posix skin variant running.
 > (my real app is based on pthread)
 > Since i only can use signals to suspend a posix thread (correct me if
 > i'm wrong) i run in the same problem with the secondary domain.
 > 
 > Is there a way to switch back to primary domain in posix skin?

In 99% of the cases, you do not need to do this, Xenomai switches
threads to primary mode when they call a service which runs in primary
mode.

For the remaining cases, there is pthread_set_mode_np, see the API
documentation at:
http://www.xenomai.org/index.php/API_documentation

To suspend threads, you can use pthread_cond_wait, sem_wait,
pthread_mutex_lock or clock_nanosleep, depending on what you are trying
to achieve. Signals are no good if you want to remain in primary mode,
because they cause a switch to secondary mode.

 > Do i have a chance to do this with pthread or do i have to rewrite my
 > code using rt_task* calls?

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-help] how to stop a runaway task

[Gilles Chanteperdrix]

Carsten Spieß wrote:
 > Hello Philippe,
 > 
 > 
 > > You will have to remove the printf() from the low prio task for the
 > > nucleus watchdog to detect the runaway situation, otherwise, your task
 > > would be considered as being under the control of the Linux scheduler
 > > until it issues a blocking Xenomai syscall (which it never does), and
 > > therefore escape the watchdog detection. That's the particular case I
 > > was talking about.
 > Meanwhile i got my sample code running:
 > When removing the printf it works fine.
 > So for me it looks like a problem when a task is switched to secondary
 > domain.
 > Is there a way to force a thread to switch back to primary domain?
 > 
 > Now i unsuccesfully tried to get a posix skin variant running.
 > (my real app is based on pthread)
 > Since i only can use signals to suspend a posix thread (correct me if
 > i'm wrong) i run in the same problem with the secondary domain.

Sorry, I wrote the previous answer without reading your previous
mails. If you are trying to suspend a task from another task, then
SIGSTOP is the only solution. Now, for the problem you have with
secondary mode switches, it is probably a bug, so you should not design
your application around this bug, it should work, and actually works on
other platforms.


-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-help] how to stop a runaway task

[Carsten Spieß]

Hello Gilles,

> If you are trying to suspend a task from another task, then
> SIGSTOP is the only solution.
O.k., this will lead me again to use rt_task_* instead of pthread_*
(i don't like signals at all)

> Now, for the problem you have with secondary mode switches,
> it is probably a bug,
Is it a known bug? Can someone confirm it? (I can provide sample code)
Is there already a fix for it?
If not, how to debug such things?
(until now, i'm only working on the user mode side)

> so you should not design  your application around this bug,
> it should work, and actually works on other platforms.

Regards Carsten
-- 
Dipl.-Ing. Carsten Spieß
Softwareentwicklung - Beratung - Schulung
Lauterlech 39d
D-86152 Augsburg
Tel.: +49(821)15 999 765
Fax.: +49(821)15 999 767
mailto://mail@domain.hid
http://www.carsten-spiess.de

Re: [Xenomai-help] how to stop a runaway task

[Gilles Chanteperdrix]

Carsten Spieß wrote:
 > Hello Gilles,
 > 
 > > If you are trying to suspend a task from another task, then
 > > SIGSTOP is the only solution.
 > O.k., this will lead me again to use rt_task_* instead of pthread_*
 > (i don't like signals at all)
 > 
 > > Now, for the problem you have with secondary mode switches,
 > > it is probably a bug,
 > Is it a known bug? Can someone confirm it? (I can provide sample code)
 > Is there already a fix for it?
 > If not, how to debug such things?
 > (until now, i'm only working on the user mode side)

Read Philippe's answer again:
https://mail.gna.org/public/xenomai-help/2007-10/msg00166.html

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-help] how to stop a runaway task

[Carsten Spieß]

 Hello Gilles,

> Read Philippe's answer again:
> https://mail.gna.org/public/xenomai-help/2007-10/msg00166.html
Hmm.., sorry, but i dont see what you mean.
Carsten
-- 
Dipl.-Ing. Carsten Spieß
Softwareentwicklung - Beratung - Schulung
Lauterlech 39d
D-86152 Augsburg
Tel.: +49(821)15 999 765
Fax.: +49(821)15 999 767
mailto://mail@domain.hid
http://www.carsten-spiess.de

Re: [Xenomai-help] how to stop a runaway task

[Gilles Chanteperdrix]

Carsten Spieß wrote:
 >  Hello Gilles,
 > 
 > > Read Philippe's answer again:
 > > https://mail.gna.org/public/xenomai-help/2007-10/msg00166.html
 > Hmm.., sorry, but i dont see what you mean.
 > Carsten

It means that the bug you observed can not be confirmed on other
platforms, so depends probably on the kernel/I-pipe patch version
combination you use, and that Philippe is trying to find a platform
where it can be reproduced so as to be able to solve it.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-help] how to stop a runaway task

[Philippe Gerum]

Carsten Spieß wrote:
> Hello,
> 
> i'm running Xenomai 2.3.2 on a PPC target with Linux 2.4.25 kernel.
> I trying to programm a supervising task, that detects a runaway task
> and suspend it.
> 
> My aproach was to make a high priority periodic task to look if
> the low prio task is running correctly and suspend it when it detects
> failure.
> But this approach didn't work, the higer prio task isn't called unless
> the lower prio task gives up cpu (by calling e.g. rt_task_yield or
> nanosleep) (see attached demo source)
> 
> When not being cooperative (removing the rt_task_yield in the demo)
> my system is reset after some seconds.
> 
> Why isn't my supervising task scheduled?
> 

This new I-pipe patch for 2.4.25 may fix the issue:
http://download.gna.org/adeos/patches/v2.4/ppc/adeos-ipipe-2.4.25-ppc-DENX-1.2-02.patch

-- 
Philippe.

Re: [Xenomai-help] how to stop a runaway task

[Carsten Spieß]

Thanks, i will try it when i´m back from holliday.
regrads Carsten
Philippe Gerum wrote:
> Carsten Spieß wrote:
>> Hello,
>>
>> i'm running Xenomai 2.3.2 on a PPC target with Linux 2.4.25 kernel.
>> I trying to programm a supervising task, that detects a runaway task
>> and suspend it.
>>
>> My aproach was to make a high priority periodic task to look if
>> the low prio task is running correctly and suspend it when it detects
>> failure.
>> But this approach didn't work, the higer prio task isn't called unless
>> the lower prio task gives up cpu (by calling e.g. rt_task_yield or
>> nanosleep) (see attached demo source)
>>
>> When not being cooperative (removing the rt_task_yield in the demo)
>> my system is reset after some seconds.
>>
>> Why isn't my supervising task scheduled?
>>
>
> This new I-pipe patch for 2.4.25 may fix the issue:
> http://download.gna.org/adeos/patches/v2.4/ppc/adeos-ipipe-2.4.25-ppc-DENX-1.2-02.patch
>
> --
> Philippe.
>
>


-- 
Carsten Spieß