[Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Sherk Chung]

[-- Attachment #1: Type: text/plain, Size: 2578 bytes --]

We are using Xenomai on an AT91 ARM board.  We wrote a program that
creates multiple Xenomai tasks, which use rt_mutexes to when accessing
some shared global variables.  The rt_mutexes used are declared
globally, as in the example below.  Since the objects sharedVar1,
shredVar2, etc. are declared on the global stack, the rt_mutexes are
created prior to main() getting executed.  The problem we are having is
that our program is causing our HW to freeze up on program load, it
never gets to the first line of main(), and our HW supplier pointed out
that we must call mlockall() and the set up the signal handlers before
creating the mutexes.

 

Is there a problem with creating rt_mutexes the way we are doing, and
should that cause the ARM board to freeze?  (the same program loads fine
on an x86)

Thanks in advance.


-Sherk

 

-------------

 

#include <native/queue.h>

#include <native/mutex.h>

#include "main.h"

 

template <class T>

class SyncObject {

    private:

RT_MUTEX mutex;

    public:

T value;

SyncObject(const char* desc){  rt_mutex_create(&mutex, desc);  }

               ~SyncObject(){  rt_mutex_delete(&mutex);  }

               void lock(){  rt_mutex_acquire(&mutex, TM_INFINITE);  } 

               void unlock(){  rt_mutex_release(&mutex);   }

} ;

 

SyncObject<int> g_sharedVar1; // global sync object used by multiple
tasks

SyncObject<int> g_sharedVar2; // global sync object used by multiple
tasks

SyncObject<int> g_sharedVar3; // global sync object used by multiple
tasks

 

main(argv, argc)

{

  printf ("Starting execution of main");   //  Program crashes before
executing this line of code

 

  signal(SIGTERM, handleShutdownSignal);

  signal(SIGINT, handleShutdownSignal);

 

  signal(SIGXCPU, handleTaskModeChange); // signal num 24

 

  mlockall(MCL_CURRENT | MCL_FUTURE);

 

  // create all queues

  err = rt_queue_create(&g_output_data_queue, g_output_data_queue_name,
OUTPUT_DATA_Q_LEN * sizeof(OutputData), OUTPUT_DATA_Q_LEN, Q_SHARED);

  err = rt_queue_create(&g_log_queue, g_log_queue_name, LOG_Q_LEN *
sizeof(LogMessage), LOG_Q_LEN, Q_SHARED);

 

  // create all remaining tasks ///////

  err = rt_task_spawn(&rtTask1,"Task1", 0, 2, 0, &task1, &errHandler1);

  err = rt_task_spawn(&rtTask2," Task2", 0, 10, 0, &task2,
&errHandler2);

  err = rt_task_spawn(&rtTask3,"Task3", 0, 10, 0, &task3, &errHandler3);

  err = rt_task_spawn(&rtTask4,"Task4", 0, 2, 0, &task4, & errHandler4);

 

}


[-- Attachment #2: Type: text/html, Size: 9175 bytes --]

Re: [Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Gilles Chanteperdrix]

Sherk Chung wrote:
> We are using Xenomai on an AT91 ARM board.  We wrote a program that
> creates multiple Xenomai tasks, which use rt_mutexes to when accessing
> some shared global variables.  The rt_mutexes used are declared
> globally, as in the example below.  Since the objects sharedVar1,
> shredVar2, etc. are declared on the global stack, the rt_mutexes are
> created prior to main() getting executed.  The problem we are having is
> that our program is causing our HW to freeze up on program load, it
> never gets to the first line of main(), and our HW supplier pointed out
> that we must call mlockall() and the set up the signal handlers before
> creating the mutexes.
> 
>  
> 
> Is there a problem with creating rt_mutexes the way we are doing, and
> should that cause the ARM board to freeze?  (the same program loads fine
> on an x86)

No, there should not be any problem. Creating a mutex does not require a
particular context, only locking it does.

Which version of Xenomai do you sue, with which version of the I-pipe patch?

-- 
					    Gilles.

Re: [Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
> Sherk Chung wrote:
>> We are using Xenomai on an AT91 ARM board.  We wrote a program that
>> creates multiple Xenomai tasks, which use rt_mutexes to when accessing
>> some shared global variables.  The rt_mutexes used are declared
>> globally, as in the example below.  Since the objects sharedVar1,
>> shredVar2, etc. are declared on the global stack, the rt_mutexes are
>> created prior to main() getting executed.  The problem we are having is
>> that our program is causing our HW to freeze up on program load, it
>> never gets to the first line of main(), and our HW supplier pointed out
>> that we must call mlockall() and the set up the signal handlers before
>> creating the mutexes.
>>
>>  
>>
>> Is there a problem with creating rt_mutexes the way we are doing, and
>> should that cause the ARM board to freeze?  (the same program loads fine
>> on an x86)
> 
> No, there should not be any problem. Creating a mutex does not require a
> particular context, only locking it does.
> 
> Which version of Xenomai do you sue, with which version of the I-pipe patch?

Is not there any output on the kernel (serial) console?


-- 
					    Gilles.

Re: [Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Travis Stratman]

Gilles,

I provided the Xenomai build for this board so I thought that it would
be appropriate for me to pick up for Sherk at this point.

On Fri, 2010-05-14 at 18:42 +0200, Gilles Chanteperdrix wrote:
> Sherk Chung wrote:
> > We are using Xenomai on an AT91 ARM board.  We wrote a program that
> > creates multiple Xenomai tasks, which use rt_mutexes to when accessing
> > some shared global variables.  The rt_mutexes used are declared
> > globally, as in the example below.  Since the objects sharedVar1,
> > shredVar2, etc. are declared on the global stack, the rt_mutexes are
> > created prior to main() getting executed.  The problem we are having is
> > that our program is causing our HW to freeze up on program load, it
> > never gets to the first line of main(), and our HW supplier pointed out
> > that we must call mlockall() and the set up the signal handlers before
> > creating the mutexes.
> > 
> >  
> > 
> > Is there a problem with creating rt_mutexes the way we are doing, and
> > should that cause the ARM board to freeze?  (the same program loads fine
> > on an x86)
> 
> No, there should not be any problem. Creating a mutex does not require a
> particular context, only locking it does.
> 
> Which version of Xenomai do you sue, with which version of the I-pipe patch?

The port is for a custom AT91-based (AT91SAM9G20) board, but we apply
the Xenomai patches directly. The kernel is 2.6.28 with the AT91 patches
and a couple of custom drivers that should not impact Xenomai. The first
build that I provided was Xenomai 2.5.2 with ipipe 1.12-07. After the
problem was reported, I tested it on a version that has been used by
several of our customers on large projects with the same board -- 2.4.8
w/ ipipe 1.12-02. I saw the same results on both.

There is no kernel output or output from running the application (ipipe
debugging is disabled). The board will not respond to any input from any
interface. 

Using strace I was able to determine that the first statements in main()
were not even reached. The strace output would generally stop on access
to /dev/rtheap or rt_sigaction(SIGXCPU...). My suspicion was the global
object instantiation calling rt_mutex_create().

Here's how I came to the conclusion that this needed to be placed inside
the application after mlockall():

First test:
I changed the global objects to pointers. The code that I was provided
had a Global.c file with these in it so I changed the code commented out
below to the pointers:
----
/*
SyncObject<int> g_terminate("terminate");
SyncObject<int> g_log_level("log level");
SyncObject<int> g_temperature("temperature");
SyncObject<SetPoint> g_set_point("Set Point");
SyncObject<int> g_decimation("decimation");
SyncObject<int> g_flow_feedback("flow feedback");
SyncObject<int> g_valve_override("valve override");
*/
SyncObject<int> * g_terminate;
SyncObject<int> * g_log_level;
SyncObject<int> * g_temperature;
SyncObject<SetPoint> * g_set_point;
SyncObject<int> * g_decimation;
SyncObject<int> * g_flow_feedback;
SyncObject<int> * g_valve_override;
----
Then, in main(), I added the following:
----

  g_terminate = new SyncObject<int>("terminate");
  g_log_level = new SyncObject<int>("log level");
  g_temperature = new SyncObject<int>("temperature");
  g_set_point = new SyncObject<SetPoint> ("Set Point");
  g_decimation = new SyncObject<int>("decimation");
  g_flow_feedback = new SyncObject<int>("flow feedback");
  g_valve_override = new SyncObject<int>("valve override");

----
This was located before mlockall() and setting up signals:
----
  signal(SIGTERM, handleShutdownSignal);
  signal(SIGINT, handleShutdownSignal);
  signal(SIGXCPU, handleTaskModeChange);
  mlockall(MCL_CURRENT | MCL_FUTURE);
----

Results:
The code always froze the board just as before. Using strace the code
would always stop output at rt_sigaction(SIGXCPU...) but it was not
actually reaching the signal() call in the application.

Second test:
I moved the object instantiation below mlockall() and the signal()
calls.

Results:
The application always completes successfully.

I can provide any other information that might be helpful. Any ideas on
what would be causing this if rt_mutex_create() can be called during the
object instantiation at a global level? Any pointers will be
appreciated!

Thanks,

TAS

Re: [Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Gilles Chanteperdrix]

Travis Stratman wrote:
> Gilles,
> 
> I provided the Xenomai build for this board so I thought that it would
> be appropriate for me to pick up for Sherk at this point.
> 
> On Fri, 2010-05-14 at 18:42 +0200, Gilles Chanteperdrix wrote:
>> Sherk Chung wrote:
>>> We are using Xenomai on an AT91 ARM board.  We wrote a program that
>>> creates multiple Xenomai tasks, which use rt_mutexes to when accessing
>>> some shared global variables.  The rt_mutexes used are declared
>>> globally, as in the example below.  Since the objects sharedVar1,
>>> shredVar2, etc. are declared on the global stack, the rt_mutexes are
>>> created prior to main() getting executed.  The problem we are having is
>>> that our program is causing our HW to freeze up on program load, it
>>> never gets to the first line of main(), and our HW supplier pointed out
>>> that we must call mlockall() and the set up the signal handlers before
>>> creating the mutexes.
>>>
>>>  
>>>
>>> Is there a problem with creating rt_mutexes the way we are doing, and
>>> should that cause the ARM board to freeze?  (the same program loads fine
>>> on an x86)
>> No, there should not be any problem. Creating a mutex does not require a
>> particular context, only locking it does.
>>
>> Which version of Xenomai do you sue, with which version of the I-pipe patch?
> 
> The port is for a custom AT91-based (AT91SAM9G20) board, but we apply
> the Xenomai patches directly. The kernel is 2.6.28 with the AT91 patches
> and a couple of custom drivers that should not impact Xenomai. The first
> build that I provided was Xenomai 2.5.2 with ipipe 1.12-07. After the
> problem was reported, I tested it on a version that has been used by
> several of our customers on large projects with the same board -- 2.4.8
> w/ ipipe 1.12-02. I saw the same results on both.
> 
> There is no kernel output or output from running the application (ipipe
> debugging is disabled). The board will not respond to any input from any
> interface. 
> 
> Using strace I was able to determine that the first statements in main()
> were not even reached. The strace output would generally stop on access
> to /dev/rtheap or rt_sigaction(SIGXCPU...). My suspicion was the global
> object instantiation calling rt_mutex_create().

I would suspect an issue with the fast mutexes, since they require a
shared heap to have been mapped prior to the mutex creation. But if you
say you saw the same issue on 2.4.8, it must be something completely
different. I will try and reproduce the issue ASAP.

-- 
					    Gilles.

Re: [Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Travis Stratman]

 
> > Using strace I was able to determine that the first statements in main()
> > were not even reached. The strace output would generally stop on access
> > to /dev/rtheap or rt_sigaction(SIGXCPU...). My suspicion was the global
> > object instantiation calling rt_mutex_create().
> 
> I would suspect an issue with the fast mutexes, since they require a
> shared heap to have been mapped prior to the mutex creation. But if you
> say you saw the same issue on 2.4.8, it must be something completely
> different. I will try and reproduce the issue ASAP.
> 

Thanks! Come to think of it I don't recall ever seeing access to rtheap
on the 2.4.8 traces, just the rt_sigaction. BTW the code is being
cross-compiled using gcc-4.2.4 arm-linux-gnueabi-g++ with -march=armv5te
-mtune=arm926ej-s flags. Let me know what else I can provide that will
be helpful.

-Travis

Re: [Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Travis Stratman]

On Fri, 2010-05-14 at 17:17 -0500, Travis Stratman wrote:
>  > > Using strace I was able to determine that the first statements in main()
> > > were not even reached. The strace output would generally stop on access
> > > to /dev/rtheap or rt_sigaction(SIGXCPU...). My suspicion was the global
> > > object instantiation calling rt_mutex_create().
> > 
> > I would suspect an issue with the fast mutexes, since they require a
> > shared heap to have been mapped prior to the mutex creation. But if you
> > say you saw the same issue on 2.4.8, it must be something completely
> > different. I will try and reproduce the issue ASAP.
> > 
> 
> Thanks! Come to think of it I don't recall ever seeing access to rtheap
> on the 2.4.8 traces, just the rt_sigaction. BTW the code is being
> cross-compiled using gcc-4.2.4 arm-linux-gnueabi-g++ with -march=armv5te
> -mtune=arm926ej-s flags. Let me know what else I can provide that will
> be helpful.

I went back and took an strace of the code that caused the board to fail
on 2.5.2 with the pointers initialized in main() but before mlockall().
This is where the strace stops:
....
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x1020000
syscall_983045(0x101fee0, 0x101fee0, 0x1020638, 0x10205b8, 0x1023058, 0, 0x58, 0xf0005, 0xffff5bfc, 0x1023058, 0x1023000, 0x1cd8b1c, 0, 0x1cd87f8, 0x8, 0x1
001ffc, 0x20000010, 0x101fee0, 0xc4a0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = 0
mprotect(0x141f000, 4096, PROT_READ)    = 0
mprotect(0x135d000, 4096, PROT_READ)    = 0
mprotect(0x134e000, 8192, PROT_READ)    = 0
....

On the second and third time that I ran after resetting it it got a
little farther:
....
mprotect(0x141f000, 4096, PROT_READ)    = 0
mprotect(0x135d000, 4096, PROT_READ)    = 0
mprotect(0x134e000, 8192, PROT_READ)    = 0
mprotect(0x1229000, 8192, PROT_READ)    = 0
mprotect(0x103e000, 4096, PROT_READ)    = 0
munmap(0x101c000, 4779)                 = 0
epoll_wait(0x101fa88, 0x1cbae14, 0x1cbae1c, 0x10279a0) = 984
ipc_subcall(0x101fa90, 0xc, 0x101faa8, 0xffffffec) = 0
rt_sigaction(SIGRTMIN, {0x1027f3c, [], SA_SIGINFO|0x4000000}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x1027e10, [], SA_RESTART|SA_SIGINFO|0x4000000}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
brk(0)                                  = 0x2c000
brk(0x4d000)                            = 0x4d000
futex(0x122f348, FUTEX_WAKE, 2147483647) = 0
rt_sigaction(SIGILL, {0x10431ec, [ILL], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGILL, {SIG_DFL}, {0x10431ec, [ILL], SA_RESTART|0x4000000}, 8) = 0
open("/dev/mem", O_RDONLY|O_SYNC)       = 3
mmap2(NULL, 4096, PROT_READ, MAP_SHARED, 3, 0xfffa0) = 0x101c000
close(3)                                = 0
open("/dev/rtheap", O_RDWR)             = 3
ioctl(3, 0, 0xc1eef410)                 = 0
mmap2(NULL, 12288, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x1421000
close(3)                                = 0
open("/dev/rtheap", O_RDWR)             = 3
ioctl(3, 0, 0xc03c5088)                 = 0
mmap2(NULL, 12288, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0)
....

The relevant code in main() looks like this:
----
  g_terminate = new SyncObject<int>("terminate");
  g_log_level = new SyncObject<int>("log level");
  g_temperature = new SyncObject<int>("temperature");
  g_set_point = new SyncObject<SetPoint> ("Set Point");
  g_decimation = new SyncObject<int>("decimation");
  g_flow_feedback = new SyncObject<int>("flow feedback");
  g_valve_override = new SyncObject<int>("valve override");

  signal(SIGTERM, handleShutdownSignal);
  signal(SIGINT, handleShutdownSignal);

  signal(SIGXCPU, handleTaskModeChange);

  mlockall(MCL_CURRENT | MCL_FUTURE);
----

I then moved the instantiation back below mlockall() and tested to make
sure that it was still working (it does). To make for a more
reproducible test I added a mutex directly in main() and tested it
before and after the mlockall(), leaving all other code as-is. i.e.:
----
RT_MUTEX test_mutex;
....
rt_mutex_create(&test_mutex, "test_mutex");
----

If rt_mutex_create() is before mlockall() the board freezes. If it is
after, the application continues. strace shows similar behavior to the
above listings.

However, when I added a similar test to a very simple timer task
application that I have, I could not reproduce the issue.

Thanks
> 
> -Travis
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help

Re: [Xenomai-help] rt_mutex created prior to main causes board to freeze?

[Gilles Chanteperdrix]

Sherk Chung wrote:
> We are using Xenomai on an AT91 ARM board.  We wrote a program that
> creates multiple Xenomai tasks, which use rt_mutexes to when accessing
> some shared global variables.  The rt_mutexes used are declared
> globally, as in the example below.  Since the objects sharedVar1,
> shredVar2, etc. are declared on the global stack, the rt_mutexes are
> created prior to main() getting executed.  The problem we are having is
> that our program is causing our HW to freeze up on program load, it
> never gets to the first line of main(), and our HW supplier pointed out
> that we must call mlockall() and the set up the signal handlers before
> creating the mutexes.
> 
>  
> 
> Is there a problem with creating rt_mutexes the way we are doing, and
> should that cause the ARM board to freeze?  (the same program loads fine
> on an x86)

Hi,

your test program does not compile, next time, please provide a 
self-contained test which I only have to compile in order to be able to 
run it. I compiled and ran the following program:

#include <native/queue.h>
#include <native/mutex.h>
#include <signal.h>
#include <sys/mman.h>


template <class T>
class SyncObject {
private:
        RT_MUTEX mutex;

public:
        T value;

        SyncObject(const char* desc)
        {
                printf("mutex_create !\n");
                rt_mutex_create(&mutex, desc);
        }

        ~SyncObject()
        {
                rt_mutex_delete(&mutex);
        }

        void lock()

        void unlock()
        {
                rt_mutex_release(&mutex);
        }
} ;

SyncObject<int> g_sharedVar1 = "var1";
SyncObject<int> g_sharedVar2 = "var2";
SyncObject<int> g_sharedVar3 = "var3";

extern "C" int main(void)
{
        printf ("Starting execution of main");   //  Program crashes before

        mlockall(MCL_CURRENT | MCL_FUTURE);
	return 0;
}

And it runs just fine. So, I need more details, such as your kernel
configuration.

-- 
					    Gilles.