[Xenomai-help] Mode switch when using RT heap on ARM

[Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

Hello,

I realized that accessing memory allocated with rt_heap_alloc() causes
mode switches on ARM i.mx31. The attached patch provides a demo program
to demonstrate the problem, which actually does *not* show up on my
PowerPC TQM5200 board.

Wolfgang.

[-- Attachment #2: example-rtheap.patch --]
[-- Type: text/x-diff, Size: 4702 bytes --]

---
 examples/native/Makefile |    4 -
 examples/native/rtheap.c |  164 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 167 insertions(+), 1 deletion(-)

Index: xenomai-2.4.x/examples/native/rtheap.c
===================================================================
--- /dev/null
+++ xenomai-2.4.x/examples/native/rtheap.c
@@ -0,0 +1,164 @@
+#include <stdio.h>
+#include <signal.h>
+#include <unistd.h>
+#include <sys/mman.h>
+
+#include <native/task.h>
+#include <native/timer.h>
+#include <native/heap.h>
+
+#define USE_HEAP
+//#define USE_SIGXCPU
+
+#define HEAP_SIZE (1024*1024)
+#define HEAP_MODE 0		/* Local heap. */
+
+RT_HEAP heap_desc;
+RT_TASK demo_task;
+
+
+static int block_sizes[] = {16, 20, 90, 150, 310, 800, 1000};
+
+/* NOTE: error handling omitted. */
+
+void demo(void *arg)
+{
+	RTIME now, previous;
+	void *block;
+	int sizes = sizeof(block_sizes) / sizeof(int);
+	int count = 0;
+	int size, err;
+	int *ptr;
+
+#ifdef USE_SIGXCPU
+	/* Ask Xenomai to warn us upon switches to secondary mode. */
+	rt_task_set_mode(0, T_WARNSW, NULL);
+#endif
+
+	/*
+	 * Arguments: &task (NULL=self),
+	 *            start time,
+	 *            period (here: 1 s)
+	 */
+	rt_task_set_periodic(NULL, TM_NOW, 10000000);
+	previous = rt_timer_read();
+
+	while (1) {
+		rt_task_wait_period(NULL);
+		now = rt_timer_read();
+
+		size = block_sizes[count % sizes];
+#ifdef USE_HEAP
+		/*
+		 * Request a 16-bytes block, asking for a non-blocking call
+		 * since only Xenomai tasks may block.
+		 */
+		err = rt_heap_alloc(&heap_desc, size, TM_NONBLOCK, &block);
+		if (err) {
+			printf("rt_heap_alloc() failed with %d\n", err);
+			break;
+		}
+
+		ptr = (int *)block;
+		*ptr = 0xdeadbeef;
+		if (*ptr != 0xdeadbeef) {
+			printf("Write/read test to heap failed\n");
+			break;
+		}
+
+		ptr = (int *)(block + size - sizeof(int));
+		*ptr = 0xbeefface;
+		if (*ptr != 0xbeefface) {
+			printf("Write/read test to heap failed\n");
+			break;
+		}
+
+		/* Free the block: */
+		rt_heap_free(&heap_desc, block);
+#endif
+
+		/*
+		 * NOTE: printf may have unexpected impact on the timing of
+		 *       your program. It is used here in the critical loop
+		 *       only for demonstration purposes.
+		 */
+		if ((count % 100) == 9) {
+			rt_printf("%d: Time since last turn: %ld.%06ld ms\n",
+				  count,
+				  (long)(now - previous) / 1000000,
+				  (long)(now - previous) % 1000000);
+			previous = now;
+		}
+		count++;
+	}
+}
+
+void catch_signal(int sig)
+{
+	printf("%s: sig=%d\n", __func__, sig);
+}
+
+#ifdef USE_SIGXCPU
+void catch_signal_msw(int sig)
+{
+	void *bt[32];
+	int nentries;
+		
+	rt_printf("%s: sig=%d\n", __func__, sig);
+
+	/* Dump a backtrace of the frame which caused the switch to
+	   secondary mode: */
+	nentries = backtrace(bt,sizeof(bt) / sizeof(bt[0]));
+	backtrace_symbols_fd(bt,nentries,fileno(stdout));
+}
+#endif
+
+int main(int argc, char* argv[])
+{
+	void *block;
+	int err;
+
+	signal(SIGTERM, catch_signal);
+	signal(SIGINT, catch_signal);
+#ifdef USE_SIGXCPU
+	signal(SIGXCPU, catch_signal_msw);
+#endif
+
+	/* Avoids memory swapping for this program */
+	mlockall(MCL_CURRENT | MCL_FUTURE);
+
+	/*
+	 * Create a 256Kb heap usable for dynamic memory allocation of
+	 * variable-size blocks in kernel space.
+	 */
+
+	err = rt_heap_create(&heap_desc, "MyHeapName", HEAP_SIZE, HEAP_MODE);
+	if (err)
+		return err;
+
+	/* Perform auto-init of rt_print buffers if the task doesn't do so */
+	rt_print_auto_init(1);
+
+	/* Initialise the rt_print buffer for this task explicitly */
+	rt_print_init(4096, "Task 1");
+
+	/*
+	 * Arguments: &task,
+	 *            name,
+	 *            stack size (0=default),
+	 *            priority,
+	 *            mode (FPU, start suspended, ...)
+	 */
+	rt_task_create(&demo_task, "trivial", 0, 99, 0);
+
+	/*
+	 * Arguments: &task,
+	 *            task function,
+	 *            function argument
+	 */
+	rt_task_start(&demo_task, &demo, NULL);
+
+	pause();
+
+	rt_task_delete(&demo_task);
+}
Index: xenomai-2.4.x/examples/native/Makefile
===================================================================
--- xenomai-2.4.x.orig/examples/native/Makefile
+++ xenomai-2.4.x/examples/native/Makefile
@@ -1,7 +1,7 @@
 ###### CONFIGURATION ######
 
 ### List of applications to be build
-APPLICATIONS = trivial-periodic sigxcpu rtprint
+APPLICATIONS = trivial-periodic sigxcpu rtprint rtheap
 
 ### Note: to override the search path for the xeno-config script, use "make XENO=..."
 
@@ -54,6 +54,8 @@ endif
 rtprint: rtprint.c
 	$(CC) $(CFLAGS) $? $(LDFLAGS) -lrtdk -o $@
 
+rtheap: rtheap.c
+	$(CC) $(CFLAGS) $? $(LDFLAGS) -lrtdk -o $@
 
 
 ###### KERNEL MODULE BUILD (no change required normally) ######

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Hello,
>
> I realized that accessing memory allocated with rt_heap_alloc() causes
> mode switches on ARM i.mx31. The attached patch provides a demo program
> to demonstrate the problem, which actually does *not* show up on my
> PowerPC TQM5200 board.

On ARM, we normally map the heaps uncacheable, this should not be
necessary on ARMv6, but I am afraid we then get the fault on first access.

-- 
                    Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> Wolfgang Grandegger wrote:
>> Hello,
>>
>> I realized that accessing memory allocated with rt_heap_alloc() causes
>> mode switches on ARM i.mx31. The attached patch provides a demo program
>> to demonstrate the problem, which actually does *not* show up on my
>> PowerPC TQM5200 board.
> 
> On ARM, we normally map the heaps uncacheable, this should not be
> necessary on ARMv6, but I am afraid we then get the fault on first access.

Is each cache-line of the heap not touched automatically when the heap
gets created? I thought it's necessary for other archs as well.

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>>> Hello,
>>>
>>> I realized that accessing memory allocated with rt_heap_alloc() causes
>>> mode switches on ARM i.mx31. The attached patch provides a demo program
>>> to demonstrate the problem, which actually does *not* show up on my
>>> PowerPC TQM5200 board.
>> On ARM, we normally map the heaps uncacheable, this should not be
>> necessary on ARMv6, but I am afraid we then get the fault on first access.
> 
> Is each cache-line of the heap not touched automatically when the heap
> gets created? I thought it's necessary for other archs as well.

s/cache-line/page/

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> Wolfgang Grandegger wrote:
>> Gilles Chanteperdrix wrote:
>>> Wolfgang Grandegger wrote:
>>>> Hello,
>>>>
>>>> I realized that accessing memory allocated with rt_heap_alloc() causes
>>>> mode switches on ARM i.mx31. The attached patch provides a demo program
>>>> to demonstrate the problem, which actually does *not* show up on my
>>>> PowerPC TQM5200 board.
>>> On ARM, we normally map the heaps uncacheable, this should not be
>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>> access.
>> Is each cache-line of the heap not touched automatically when the heap
>> gets created? I thought it's necessary for other archs as well.
> 
> No, the thing which comes near to this is the workaround in I-pipe of the
> way pages are write-protected upon fork. But this happens only upon fork.
> I am not sure I understand all the subtleties of ARM memory management,
> but I think this fault on first write is the way the "dirty" bit is
> implemented.

Well, it seems not to be that simple. My attached rtheap example program
behaves the following way:

- Heap-mode=0: I see plenty of mode switches also for read-only:

  # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
  CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
    0  0      0          2951       0     00400080   99.7  ROOT
    0  929    102        398        3     00300184    0.0  rtheap
    0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
  CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
    0  0      0          4287       0     00400080   99.4  ROOT
    0  929    435        1734       3     00300184    0.4  rtheap
    0  0      0          25010      0     00000000    0.1  IRQ29: [timer]

- Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
  gets very slow:

  # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
  CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
    0  0      0          5358       0     00400080  100.0  ROOT
    0  941    2          5          1     00300380    0.0  rtheap
    0  0      0          39030      0     00000000    0.0  IRQ29: [timer]
  CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
    0  0      0          5358       0     00400080   99.9  ROOT
    0  941    2          5          1     00300380    0.0  rtheap
    0  0      0          41232      0     00000000    0.1  IRQ29: [timer]

Hm, am I doing something wrong?

Wolfgang.



#include <stdio.h>
#include <signal.h>
#include <unistd.h>
#include <sys/mman.h>

#include <native/task.h>
#include <native/timer.h>
#include <native/heap.h>

#define USE_HEAP
//#define USE_SIGXCPU

#define HEAP_SIZE (1024*1024)
#if 1
#define HEAP_MODE 0		/* Local heap. */
#else
#define HEAP_MODE H_NONCACHED
#endif

RT_HEAP heap_desc;
RT_TASK demo_task;


static int block_sizes[] = {16, 20, 90, 150, 310, 800, 1000, 5000, 9000};

/* NOTE: error handling omitted. */

void demo(void *arg)
{
	RTIME now, previous;
	void *block;
	int sizes = sizeof(block_sizes) / sizeof(int);
	int count = 0;
	int size, err;
	int *ptr;

#ifdef USE_SIGXCPU
	/* Ask Xenomai to warn us upon switches to secondary mode. */
	rt_task_set_mode(0, T_WARNSW, NULL);
#endif

	/*
	 * Arguments: &task (NULL=self),
	 *            start time,
	 *            period (here: 1 s)
	 */
	rt_task_set_periodic(NULL, TM_NOW, 10000000);
	previous = rt_timer_read();

	while (1) {
		rt_task_wait_period(NULL);
		now = rt_timer_read();

		size = block_sizes[count % sizes];
#ifdef USE_HEAP
		/*
		 * Request a 16-bytes block, asking for a non-blocking call
		 * since only Xenomai tasks may block.
		 */
		err = rt_heap_alloc(&heap_desc, size, TM_NONBLOCK, &block);
		if (err) {
			printf("rt_heap_alloc() failed with %d\n", err);
			break;
		}

		ptr = (int *)block;
		*ptr = 0xdeadbeef;
		if (*ptr != 0xdeadbeef) {
			printf("Write/read test to heap failed\n");
			break;
		}

		ptr = (int *)(block + size - sizeof(int));
		*ptr = 0xbeefface;
		if (*ptr != 0xbeefface) {
			printf("Write/read test to heap failed\n");
			break;
		}

		/* Free the block: */
		rt_heap_free(&heap_desc, block);
#endif

		/*
		 * NOTE: printf may have unexpected impact on the timing of
		 *       your program. It is used here in the critical loop
		 *       only for demonstration purposes.
		 */
		if ((count % 100) == 9) {
			rt_printf("%d: Time since last turn: %ld.%06ld ms\n",
				  count,
				  (long)(now - previous) / 1000000,
				  (long)(now - previous) % 1000000);
			previous = now;
		}
		count++;
	}
}

void catch_signal(int sig)
{
	printf("%s: sig=%d\n", __func__, sig);
}

#ifdef USE_SIGXCPU
void catch_signal_msw(int sig)
{
	void *bt[32];
	int nentries;
		
	rt_printf("%s: sig=%d\n", __func__, sig);

	/* Dump a backtrace of the frame which caused the switch to
	   secondary mode: */
	nentries = backtrace(bt,sizeof(bt) / sizeof(bt[0]));
	backtrace_symbols_fd(bt,nentries,fileno(stdout));
}
#endif

int main(int argc, char* argv[])
{
	void *block;
	int err;

	signal(SIGTERM, catch_signal);
	signal(SIGINT, catch_signal);
#ifdef USE_SIGXCPU
	signal(SIGXCPU, catch_signal_msw);
#endif

	/* Avoids memory swapping for this program */
	mlockall(MCL_CURRENT | MCL_FUTURE);

	/*
	 * Create a 256Kb heap usable for dynamic memory allocation of
	 * variable-size blocks in kernel space.
	 */

	err = rt_heap_create(&heap_desc, "MyHeapName", HEAP_SIZE, HEAP_MODE);
	if (err)
		return err;

	/* Perform auto-init of rt_print buffers if the task doesn't do so */
	rt_print_auto_init(1);

	/* Initialise the rt_print buffer for this task explicitly */
	rt_print_init(4096, "Task 1");

	/*
	 * Arguments: &task,
	 *            name,
	 *            stack size (0=default),
	 *            priority,
	 *            mode (FPU, start suspended, ...)
	 */
	rt_task_create(&demo_task, "rtheap", 0, 99, 0);

	/*
	 * Arguments: &task,
	 *            task function,
	 *            function argument
	 */
	rt_task_start(&demo_task, &demo, NULL);

	pause();

	rt_task_delete(&demo_task);
}

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Wolfgang Grandegger wrote:
>>>>> Hello,
>>>>>
>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>> causes
>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>> program
>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>> PowerPC TQM5200 board.
>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>> access.
>>> Is each cache-line of the heap not touched automatically when the heap
>>> gets created? I thought it's necessary for other archs as well.
>>
>> No, the thing which comes near to this is the workaround in I-pipe of
>> the
>> way pages are write-protected upon fork. But this happens only upon
>> fork.
>> I am not sure I understand all the subtleties of ARM memory management,
>> but I think this fault on first write is the way the "dirty" bit is
>> implemented.
>
> Well, it seems not to be that simple. My attached rtheap example program
> behaves the following way:
>
> - Heap-mode=0: I see plenty of mode switches also for read-only:

Maybe that is expected on ARM ? But what I do not understand is that you
get 300 modes switches where as your example makes two faults every 10ms
during 10 seconds, so, you should see something like 1000 modes switches.

>
>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>     0  0      0          2951       0     00400080   99.7  ROOT
>     0  929    102        398        3     00300184    0.0  rtheap
>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>     0  0      0          4287       0     00400080   99.4  ROOT
>     0  929    435        1734       3     00300184    0.4  rtheap
>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>
> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>   gets very slow:

That is expected. But where do you get the 2 mode switches ?

-- 
                    Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> Wolfgang Grandegger wrote:
>> Gilles Chanteperdrix wrote:
>>> Wolfgang Grandegger wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Wolfgang Grandegger wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>>> causes
>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>>> program
>>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>>> PowerPC TQM5200 board.
>>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>>> access.
>>>> Is each cache-line of the heap not touched automatically when the heap
>>>> gets created? I thought it's necessary for other archs as well.
>>> No, the thing which comes near to this is the workaround in I-pipe of
>>> the
>>> way pages are write-protected upon fork. But this happens only upon
>>> fork.
>>> I am not sure I understand all the subtleties of ARM memory management,
>>> but I think this fault on first write is the way the "dirty" bit is
>>> implemented.
>> Well, it seems not to be that simple. My attached rtheap example program
>> behaves the following way:
>>
>> - Heap-mode=0: I see plenty of mode switches also for read-only:
> 
> Maybe that is expected on ARM ? But what I do not understand is that you
> get 300 modes switches where as your example makes two faults every 10ms
> during 10 seconds, so, you should see something like 1000 modes switches.

And even more because the program writes/reads at the beginning and end
of the buffer. I realized that as well. Not every read/write seems to
provoke a mode switch.

>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>     0  0      0          2951       0     00400080   99.7  ROOT
>>     0  929    102        398        3     00300184    0.0  rtheap
>>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>     0  0      0          4287       0     00400080   99.4  ROOT
>>     0  929    435        1734       3     00300184    0.4  rtheap
>>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>>
>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>   gets very slow:
> 
> That is expected. But where do you get the 2 mode switches ?

When the task writes to the heap memory for the first time. And I just
realized that the write/read to the end of the buffer makes real
trouble. The task seems to hang (or wait for something):

-bash-3.2# cat sched
CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
  0  0       99      0          0          master    R          ROOT
  0  1114    99      100000000  0          master    X          rtheap

At the same time the system gets slow.

So far I understood from your comments that rtheap is not really usable
on ARM, right? What other option do I have?

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Wolfgang Grandegger wrote:
>>>>> Gilles Chanteperdrix wrote:
>>>>>> Wolfgang Grandegger wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>>>> causes
>>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>>>> program
>>>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>>>> PowerPC TQM5200 board.
>>>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>>>> access.
>>>>> Is each cache-line of the heap not touched automatically when the heap
>>>>> gets created? I thought it's necessary for other archs as well.
>>>> No, the thing which comes near to this is the workaround in I-pipe of
>>>> the
>>>> way pages are write-protected upon fork. But this happens only upon
>>>> fork.
>>>> I am not sure I understand all the subtleties of ARM memory management,
>>>> but I think this fault on first write is the way the "dirty" bit is
>>>> implemented.
>>> Well, it seems not to be that simple. My attached rtheap example program
>>> behaves the following way:
>>>
>>> - Heap-mode=0: I see plenty of mode switches also for read-only:
>> Maybe that is expected on ARM ? But what I do not understand is that you
>> get 300 modes switches where as your example makes two faults every 10ms
>> during 10 seconds, so, you should see something like 1000 modes switches.
> 
> And even more because the program writes/reads at the beginning and end
> of the buffer. I realized that as well. Not every read/write seems to
> provoke a mode switch.
> 
>>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>     0  0      0          2951       0     00400080   99.7  ROOT
>>>     0  929    102        398        3     00300184    0.0  rtheap
>>>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>     0  0      0          4287       0     00400080   99.4  ROOT
>>>     0  929    435        1734       3     00300184    0.4  rtheap
>>>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>>>
>>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>>   gets very slow:
>> That is expected. But where do you get the 2 mode switches ?
> 
> When the task writes to the heap memory for the first time. And I just
> realized that the write/read to the end of the buffer makes real
> trouble. The task seems to hang (or wait for something):
> 
> -bash-3.2# cat sched
> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>   0  0       99      0          0          master    R          ROOT
>   0  1114    99      100000000  0          master    X          rtheap
> 
> At the same time the system gets slow.
> 
> So far I understood from your comments that rtheap is not really usable
> on ARM, right? What other option do I have?

I need to double check, but I am almost sure that we do not get faults
with uncacheable heaps on ARM < 6, because we use such heaps for fast
mutexes.

So, what you observe is probably an ARMv6 or VIPT cache effect.

Now, from your program, it seems that you use rtheaps as real-time
allocator. This is overkill, rtheaps are designed to share memory
between kernel-space and user-space. For other usages, you probably do
not need rtheaps.

Nevertheless, I agree that they need fixing on ARMv6.

-- 
					    Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> Wolfgang Grandegger wrote:
>> Gilles Chanteperdrix wrote:
>>> Wolfgang Grandegger wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Wolfgang Grandegger wrote:
>>>>>> Gilles Chanteperdrix wrote:
>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>>>>> causes
>>>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>>>>> program
>>>>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>>>>> PowerPC TQM5200 board.
>>>>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>>>>> access.
>>>>>> Is each cache-line of the heap not touched automatically when the heap
>>>>>> gets created? I thought it's necessary for other archs as well.
>>>>> No, the thing which comes near to this is the workaround in I-pipe of
>>>>> the
>>>>> way pages are write-protected upon fork. But this happens only upon
>>>>> fork.
>>>>> I am not sure I understand all the subtleties of ARM memory management,
>>>>> but I think this fault on first write is the way the "dirty" bit is
>>>>> implemented.
>>>> Well, it seems not to be that simple. My attached rtheap example program
>>>> behaves the following way:
>>>>
>>>> - Heap-mode=0: I see plenty of mode switches also for read-only:
>>> Maybe that is expected on ARM ? But what I do not understand is that you
>>> get 300 modes switches where as your example makes two faults every 10ms
>>> during 10 seconds, so, you should see something like 1000 modes switches.
>> And even more because the program writes/reads at the beginning and end
>> of the buffer. I realized that as well. Not every read/write seems to
>> provoke a mode switch.
>>
>>>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>     0  0      0          2951       0     00400080   99.7  ROOT
>>>>     0  929    102        398        3     00300184    0.0  rtheap
>>>>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>     0  0      0          4287       0     00400080   99.4  ROOT
>>>>     0  929    435        1734       3     00300184    0.4  rtheap
>>>>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>>>>
>>>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>>>   gets very slow:
>>> That is expected. But where do you get the 2 mode switches ?
>> When the task writes to the heap memory for the first time. And I just
>> realized that the write/read to the end of the buffer makes real
>> trouble. The task seems to hang (or wait for something):
>>
>> -bash-3.2# cat sched
>> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>>   0  0       99      0          0          master    R          ROOT
>>   0  1114    99      100000000  0          master    X          rtheap
>>
>> At the same time the system gets slow.
>>
>> So far I understood from your comments that rtheap is not really usable
>> on ARM, right? What other option do I have?
> 
> I need to double check, but I am almost sure that we do not get faults
> with uncacheable heaps on ARM < 6, because we use such heaps for fast
> mutexes.
> 
> So, what you observe is probably an ARMv6 or VIPT cache effect.
> 
> Now, from your program, it seems that you use rtheaps as real-time
> allocator. This is overkill, rtheaps are designed to share memory
> between kernel-space and user-space. For other usages, you probably do
> not need rtheaps.

I need to dynamically allocate and free memory in a real-time task. I
thought that the RT heap is exactly for that purpose but that seems not
to be case. I actually need to implement my own simply allocater using
malloc'ed memory.

> Nevertheless, I agree that they need fixing on ARMv6.

Looks like.

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Philippe Gerum]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Wolfgang Grandegger wrote:
>>>>> Gilles Chanteperdrix wrote:
>>>>>> Wolfgang Grandegger wrote:
>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>>>>>> causes
>>>>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>>>>>> program
>>>>>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>>>>>> PowerPC TQM5200 board.
>>>>>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>>>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>>>>>> access.
>>>>>>> Is each cache-line of the heap not touched automatically when the heap
>>>>>>> gets created? I thought it's necessary for other archs as well.
>>>>>> No, the thing which comes near to this is the workaround in I-pipe of
>>>>>> the
>>>>>> way pages are write-protected upon fork. But this happens only upon
>>>>>> fork.
>>>>>> I am not sure I understand all the subtleties of ARM memory management,
>>>>>> but I think this fault on first write is the way the "dirty" bit is
>>>>>> implemented.
>>>>> Well, it seems not to be that simple. My attached rtheap example program
>>>>> behaves the following way:
>>>>>
>>>>> - Heap-mode=0: I see plenty of mode switches also for read-only:
>>>> Maybe that is expected on ARM ? But what I do not understand is that you
>>>> get 300 modes switches where as your example makes two faults every 10ms
>>>> during 10 seconds, so, you should see something like 1000 modes switches.
>>> And even more because the program writes/reads at the beginning and end
>>> of the buffer. I realized that as well. Not every read/write seems to
>>> provoke a mode switch.
>>>
>>>>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>     0  0      0          2951       0     00400080   99.7  ROOT
>>>>>     0  929    102        398        3     00300184    0.0  rtheap
>>>>>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>     0  0      0          4287       0     00400080   99.4  ROOT
>>>>>     0  929    435        1734       3     00300184    0.4  rtheap
>>>>>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>>>>>
>>>>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>>>>   gets very slow:
>>>> That is expected. But where do you get the 2 mode switches ?
>>> When the task writes to the heap memory for the first time. And I just
>>> realized that the write/read to the end of the buffer makes real
>>> trouble. The task seems to hang (or wait for something):
>>>
>>> -bash-3.2# cat sched
>>> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>>>   0  0       99      0          0          master    R          ROOT
>>>   0  1114    99      100000000  0          master    X          rtheap
>>>
>>> At the same time the system gets slow.
>>>
>>> So far I understood from your comments that rtheap is not really usable
>>> on ARM, right? What other option do I have?
>> I need to double check, but I am almost sure that we do not get faults
>> with uncacheable heaps on ARM < 6, because we use such heaps for fast
>> mutexes.
>>
>> So, what you observe is probably an ARMv6 or VIPT cache effect.
>>
>> Now, from your program, it seems that you use rtheaps as real-time
>> allocator. This is overkill, rtheaps are designed to share memory
>> between kernel-space and user-space. For other usages, you probably do
>> not need rtheaps.
> 
> I need to dynamically allocate and free memory in a real-time task. I
> thought that the RT heap is exactly for that purpose but that seems not
> to be case.

It is also usable that way when H_SINGLE is unset in the creation mode; until we
had fast synch objects in user-space, the cost of issuing a single syscall to
get a memory chunk from kernel space was certainly lower than issuing mutex lock
and unlock syscalls, as one would have to in order to implement the allocator
fully in userland.

This may change with the upcoming 2.5 which introduces fast synchs though, but
obviously, treading on the underlying memory should not cause bad side-effects
like unwanted mode switches in the first place. For the rest, we could rebase
the TLSF allocator on fast synchs in 2.5, and provide this as part of the rtdk,
but we would need a nucleus-based, skin-agnostic interface to fast synchs as well.

 I actually need to implement my own simply allocater using
> malloc'ed memory.
> 
>> Nevertheless, I agree that they need fixing on ARMv6.
> 
> Looks like.
> 
> Wolfgang.
> 
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
> 


-- 
Philippe.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> Wolfgang Grandegger wrote:
>> Gilles Chanteperdrix wrote:
>>> Wolfgang Grandegger wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Wolfgang Grandegger wrote:
>>>>>> Gilles Chanteperdrix wrote:
>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>>>>>>> causes
>>>>>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>>>>>>> program
>>>>>>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>>>>>>> PowerPC TQM5200 board.
>>>>>>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>>>>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>>>>>>> access.
>>>>>>>> Is each cache-line of the heap not touched automatically when the heap
>>>>>>>> gets created? I thought it's necessary for other archs as well.
>>>>>>> No, the thing which comes near to this is the workaround in I-pipe of
>>>>>>> the
>>>>>>> way pages are write-protected upon fork. But this happens only upon
>>>>>>> fork.
>>>>>>> I am not sure I understand all the subtleties of ARM memory management,
>>>>>>> but I think this fault on first write is the way the "dirty" bit is
>>>>>>> implemented.
>>>>>> Well, it seems not to be that simple. My attached rtheap example program
>>>>>> behaves the following way:
>>>>>>
>>>>>> - Heap-mode=0: I see plenty of mode switches also for read-only:
>>>>> Maybe that is expected on ARM ? But what I do not understand is that you
>>>>> get 300 modes switches where as your example makes two faults every 10ms
>>>>> during 10 seconds, so, you should see something like 1000 modes switches.
>>>> And even more because the program writes/reads at the beginning and end
>>>> of the buffer. I realized that as well. Not every read/write seems to
>>>> provoke a mode switch.
>>>>
>>>>>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>>     0  0      0          2951       0     00400080   99.7  ROOT
>>>>>>     0  929    102        398        3     00300184    0.0  rtheap
>>>>>>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>>     0  0      0          4287       0     00400080   99.4  ROOT
>>>>>>     0  929    435        1734       3     00300184    0.4  rtheap
>>>>>>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>>>>>>
>>>>>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>>>>>   gets very slow:
>>>>> That is expected. But where do you get the 2 mode switches ?
>>>> When the task writes to the heap memory for the first time. And I just
>>>> realized that the write/read to the end of the buffer makes real
>>>> trouble. The task seems to hang (or wait for something):
>>>>
>>>> -bash-3.2# cat sched
>>>> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>>>>   0  0       99      0          0          master    R          ROOT
>>>>   0  1114    99      100000000  0          master    X          rtheap
>>>>
>>>> At the same time the system gets slow.
>>>>
>>>> So far I understood from your comments that rtheap is not really usable
>>>> on ARM, right? What other option do I have?
>>> I need to double check, but I am almost sure that we do not get faults
>>> with uncacheable heaps on ARM < 6, because we use such heaps for fast
>>> mutexes.
>>>
>>> So, what you observe is probably an ARMv6 or VIPT cache effect.
>>>
>>> Now, from your program, it seems that you use rtheaps as real-time
>>> allocator. This is overkill, rtheaps are designed to share memory
>>> between kernel-space and user-space. For other usages, you probably do
>>> not need rtheaps.
>> I need to dynamically allocate and free memory in a real-time task. I
>> thought that the RT heap is exactly for that purpose but that seems not
>> to be case.
> 
> It is also usable that way when H_SINGLE is unset in the creation mode; until we
> had fast synch objects in user-space, the cost of issuing a single syscall to
> get a memory chunk from kernel space was certainly lower than issuing mutex lock
> and unlock syscalls, as one would have to in order to implement the allocator
> fully in userland.

But maybe in this case, we should use some memory from an anonymous
mapping, instead of using memory from a shared heap ?

-- 
                                                 Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Philippe Gerum wrote:
> Wolfgang Grandegger wrote:
>> Gilles Chanteperdrix wrote:
>>> Wolfgang Grandegger wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Wolfgang Grandegger wrote:
>>>>>> Gilles Chanteperdrix wrote:
>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>>>>>>> causes
>>>>>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>>>>>>> program
>>>>>>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>>>>>>> PowerPC TQM5200 board.
>>>>>>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>>>>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>>>>>>> access.
>>>>>>>> Is each cache-line of the heap not touched automatically when the heap
>>>>>>>> gets created? I thought it's necessary for other archs as well.
>>>>>>> No, the thing which comes near to this is the workaround in I-pipe of
>>>>>>> the
>>>>>>> way pages are write-protected upon fork. But this happens only upon
>>>>>>> fork.
>>>>>>> I am not sure I understand all the subtleties of ARM memory management,
>>>>>>> but I think this fault on first write is the way the "dirty" bit is
>>>>>>> implemented.
>>>>>> Well, it seems not to be that simple. My attached rtheap example program
>>>>>> behaves the following way:
>>>>>>
>>>>>> - Heap-mode=0: I see plenty of mode switches also for read-only:
>>>>> Maybe that is expected on ARM ? But what I do not understand is that you
>>>>> get 300 modes switches where as your example makes two faults every 10ms
>>>>> during 10 seconds, so, you should see something like 1000 modes switches.
>>>> And even more because the program writes/reads at the beginning and end
>>>> of the buffer. I realized that as well. Not every read/write seems to
>>>> provoke a mode switch.
>>>>
>>>>>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>>     0  0      0          2951       0     00400080   99.7  ROOT
>>>>>>     0  929    102        398        3     00300184    0.0  rtheap
>>>>>>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>>     0  0      0          4287       0     00400080   99.4  ROOT
>>>>>>     0  929    435        1734       3     00300184    0.4  rtheap
>>>>>>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>>>>>>
>>>>>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>>>>>   gets very slow:
>>>>> That is expected. But where do you get the 2 mode switches ?
>>>> When the task writes to the heap memory for the first time. And I just
>>>> realized that the write/read to the end of the buffer makes real
>>>> trouble. The task seems to hang (or wait for something):
>>>>
>>>> -bash-3.2# cat sched
>>>> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>>>>   0  0       99      0          0          master    R          ROOT
>>>>   0  1114    99      100000000  0          master    X          rtheap
>>>>
>>>> At the same time the system gets slow.
>>>>
>>>> So far I understood from your comments that rtheap is not really usable
>>>> on ARM, right? What other option do I have?
>>> I need to double check, but I am almost sure that we do not get faults
>>> with uncacheable heaps on ARM < 6, because we use such heaps for fast
>>> mutexes.
>>>
>>> So, what you observe is probably an ARMv6 or VIPT cache effect.
>>>
>>> Now, from your program, it seems that you use rtheaps as real-time
>>> allocator. This is overkill, rtheaps are designed to share memory
>>> between kernel-space and user-space. For other usages, you probably do
>>> not need rtheaps.
>> I need to dynamically allocate and free memory in a real-time task. I
>> thought that the RT heap is exactly for that purpose but that seems not
>> to be case.
> 
> It is also usable that way when H_SINGLE is unset in the creation mode; until we
> had fast synch objects in user-space, the cost of issuing a single syscall to
> get a memory chunk from kernel space was certainly lower than issuing mutex lock
> and unlock syscalls, as one would have to in order to implement the allocator
> fully in userland.

OK, on ARM things seems to be worse because slow uncachable memory needs
to be used.

> This may change with the upcoming 2.5 which introduces fast synchs though, but
> obviously, treading on the underlying memory should not cause bad side-effects
> like unwanted mode switches in the first place. For the rest, we could rebase
> the TLSF allocator on fast synchs in 2.5, and provide this as part of the rtdk,
> but we would need a nucleus-based, skin-agnostic interface to fast synchs as well.

Sounds good. I vote for it. I remember that Jan posted a TLSF allocator
for the nucleus some time ago. Either I port it to user-space or I will
use a simple allocater like RTnet's alloc_rtskb() for the time being.

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Philippe Gerum]

Wolfgang Grandegger wrote:
> Philippe Gerum wrote:
>> Wolfgang Grandegger wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Wolfgang Grandegger wrote:
>>>>> Gilles Chanteperdrix wrote:
>>>>>> Wolfgang Grandegger wrote:
>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>>>>>>>> causes
>>>>>>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>>>>>>>> program
>>>>>>>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>>>>>>>> PowerPC TQM5200 board.
>>>>>>>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>>>>>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>>>>>>>> access.
>>>>>>>>> Is each cache-line of the heap not touched automatically when the heap
>>>>>>>>> gets created? I thought it's necessary for other archs as well.
>>>>>>>> No, the thing which comes near to this is the workaround in I-pipe of
>>>>>>>> the
>>>>>>>> way pages are write-protected upon fork. But this happens only upon
>>>>>>>> fork.
>>>>>>>> I am not sure I understand all the subtleties of ARM memory management,
>>>>>>>> but I think this fault on first write is the way the "dirty" bit is
>>>>>>>> implemented.
>>>>>>> Well, it seems not to be that simple. My attached rtheap example program
>>>>>>> behaves the following way:
>>>>>>>
>>>>>>> - Heap-mode=0: I see plenty of mode switches also for read-only:
>>>>>> Maybe that is expected on ARM ? But what I do not understand is that you
>>>>>> get 300 modes switches where as your example makes two faults every 10ms
>>>>>> during 10 seconds, so, you should see something like 1000 modes switches.
>>>>> And even more because the program writes/reads at the beginning and end
>>>>> of the buffer. I realized that as well. Not every read/write seems to
>>>>> provoke a mode switch.
>>>>>
>>>>>>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>>>     0  0      0          2951       0     00400080   99.7  ROOT
>>>>>>>     0  929    102        398        3     00300184    0.0  rtheap
>>>>>>>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>>>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>>>     0  0      0          4287       0     00400080   99.4  ROOT
>>>>>>>     0  929    435        1734       3     00300184    0.4  rtheap
>>>>>>>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>>>>>>>
>>>>>>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>>>>>>   gets very slow:
>>>>>> That is expected. But where do you get the 2 mode switches ?
>>>>> When the task writes to the heap memory for the first time. And I just
>>>>> realized that the write/read to the end of the buffer makes real
>>>>> trouble. The task seems to hang (or wait for something):
>>>>>
>>>>> -bash-3.2# cat sched
>>>>> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>>>>>   0  0       99      0          0          master    R          ROOT
>>>>>   0  1114    99      100000000  0          master    X          rtheap
>>>>>
>>>>> At the same time the system gets slow.
>>>>>
>>>>> So far I understood from your comments that rtheap is not really usable
>>>>> on ARM, right? What other option do I have?
>>>> I need to double check, but I am almost sure that we do not get faults
>>>> with uncacheable heaps on ARM < 6, because we use such heaps for fast
>>>> mutexes.
>>>>
>>>> So, what you observe is probably an ARMv6 or VIPT cache effect.
>>>>
>>>> Now, from your program, it seems that you use rtheaps as real-time
>>>> allocator. This is overkill, rtheaps are designed to share memory
>>>> between kernel-space and user-space. For other usages, you probably do
>>>> not need rtheaps.
>>> I need to dynamically allocate and free memory in a real-time task. I
>>> thought that the RT heap is exactly for that purpose but that seems not
>>> to be case.
>> It is also usable that way when H_SINGLE is unset in the creation mode; until we
>> had fast synch objects in user-space, the cost of issuing a single syscall to
>> get a memory chunk from kernel space was certainly lower than issuing mutex lock
>> and unlock syscalls, as one would have to in order to implement the allocator
>> fully in userland.
> 
> OK, on ARM things seems to be worse because slow uncachable memory needs
> to be used.
> 
>> This may change with the upcoming 2.5 which introduces fast synchs though, but
>> obviously, treading on the underlying memory should not cause bad side-effects
>> like unwanted mode switches in the first place. For the rest, we could rebase
>> the TLSF allocator on fast synchs in 2.5, and provide this as part of the rtdk,
>> but we would need a nucleus-based, skin-agnostic interface to fast synchs as well.
> 
> Sounds good. I vote for it. I remember that Jan posted a TLSF allocator
> for the nucleus some time ago. Either I port it to user-space or I will
> use a simple allocater like RTnet's alloc_rtskb() for the time being.
>

You could pick the TLSF port I made for Xenomai/SOLO as well, it only requires
you to wrap the mutex-related calls to the proper native skin support.

> Wolfgang.
> 


-- 
Philippe.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Philippe Gerum wrote:
> Wolfgang Grandegger wrote:
>> Philippe Gerum wrote:
>>> Wolfgang Grandegger wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Wolfgang Grandegger wrote:
>>>>>> Gilles Chanteperdrix wrote:
>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>>>>> Wolfgang Grandegger wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
>>>>>>>>>>>> causes
>>>>>>>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
>>>>>>>>>>>> program
>>>>>>>>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>>>>>>>>> PowerPC TQM5200 board.
>>>>>>>>>>> On ARM, we normally map the heaps uncacheable, this should not be
>>>>>>>>>>> necessary on ARMv6, but I am afraid we then get the fault on first
>>>>>>>>>>> access.
>>>>>>>>>> Is each cache-line of the heap not touched automatically when the heap
>>>>>>>>>> gets created? I thought it's necessary for other archs as well.
>>>>>>>>> No, the thing which comes near to this is the workaround in I-pipe of
>>>>>>>>> the
>>>>>>>>> way pages are write-protected upon fork. But this happens only upon
>>>>>>>>> fork.
>>>>>>>>> I am not sure I understand all the subtleties of ARM memory management,
>>>>>>>>> but I think this fault on first write is the way the "dirty" bit is
>>>>>>>>> implemented.
>>>>>>>> Well, it seems not to be that simple. My attached rtheap example program
>>>>>>>> behaves the following way:
>>>>>>>>
>>>>>>>> - Heap-mode=0: I see plenty of mode switches also for read-only:
>>>>>>> Maybe that is expected on ARM ? But what I do not understand is that you
>>>>>>> get 300 modes switches where as your example makes two faults every 10ms
>>>>>>> during 10 seconds, so, you should see something like 1000 modes switches.
>>>>>> And even more because the program writes/reads at the beginning and end
>>>>>> of the buffer. I realized that as well. Not every read/write seems to
>>>>>> provoke a mode switch.
>>>>>>
>>>>>>>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>>>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>>>>     0  0      0          2951       0     00400080   99.7  ROOT
>>>>>>>>     0  929    102        398        3     00300184    0.0  rtheap
>>>>>>>>     0  0      0          23005      0     00000000    0.1  IRQ29: [timer]
>>>>>>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>>>>>>     0  0      0          4287       0     00400080   99.4  ROOT
>>>>>>>>     0  929    435        1734       3     00300184    0.4  rtheap
>>>>>>>>     0  0      0          25010      0     00000000    0.1  IRQ29: [timer]
>>>>>>>>
>>>>>>>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>>>>>>>   gets very slow:
>>>>>>> That is expected. But where do you get the 2 mode switches ?
>>>>>> When the task writes to the heap memory for the first time. And I just
>>>>>> realized that the write/read to the end of the buffer makes real
>>>>>> trouble. The task seems to hang (or wait for something):
>>>>>>
>>>>>> -bash-3.2# cat sched
>>>>>> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>>>>>>   0  0       99      0          0          master    R          ROOT
>>>>>>   0  1114    99      100000000  0          master    X          rtheap
>>>>>>
>>>>>> At the same time the system gets slow.
>>>>>>
>>>>>> So far I understood from your comments that rtheap is not really usable
>>>>>> on ARM, right? What other option do I have?
>>>>> I need to double check, but I am almost sure that we do not get faults
>>>>> with uncacheable heaps on ARM < 6, because we use such heaps for fast
>>>>> mutexes.
>>>>>
>>>>> So, what you observe is probably an ARMv6 or VIPT cache effect.
>>>>>
>>>>> Now, from your program, it seems that you use rtheaps as real-time
>>>>> allocator. This is overkill, rtheaps are designed to share memory
>>>>> between kernel-space and user-space. For other usages, you probably do
>>>>> not need rtheaps.
>>>> I need to dynamically allocate and free memory in a real-time task. I
>>>> thought that the RT heap is exactly for that purpose but that seems not
>>>> to be case.
>>> It is also usable that way when H_SINGLE is unset in the creation mode; until we
>>> had fast synch objects in user-space, the cost of issuing a single syscall to
>>> get a memory chunk from kernel space was certainly lower than issuing mutex lock
>>> and unlock syscalls, as one would have to in order to implement the allocator
>>> fully in userland.
>> OK, on ARM things seems to be worse because slow uncachable memory needs
>> to be used.
>>
>>> This may change with the upcoming 2.5 which introduces fast synchs though, but
>>> obviously, treading on the underlying memory should not cause bad side-effects
>>> like unwanted mode switches in the first place. For the rest, we could rebase
>>> the TLSF allocator on fast synchs in 2.5, and provide this as part of the rtdk,
>>> but we would need a nucleus-based, skin-agnostic interface to fast synchs as well.
>> Sounds good. I vote for it. I remember that Jan posted a TLSF allocator
>> for the nucleus some time ago. Either I port it to user-space or I will
>> use a simple allocater like RTnet's alloc_rtskb() for the time being.
>>
> 
> You could pick the TLSF port I made for Xenomai/SOLO as well, it only requires
> you to wrap the mutex-related calls to the proper native skin support.

Ah, cool.

Thanks for the hint.

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Hello,
> 
> I realized that accessing memory allocated with rt_heap_alloc() causes
> mode switches on ARM i.mx31. The attached patch provides a demo program
> to demonstrate the problem, which actually does *not* show up on my
> PowerPC TQM5200 board.

Ok. I ran your test on an ARMv5 using an old version of Xenomai, and I
only get a fault on first write to the heap, being either cached or non
cached.

I will now run the tests on AT91RM9200 with a more recent version of
xenomai to confirm that we get the same effect, but if we do, this means
that we are facing an armv6 specific issue and that for a workaround, we
need to understand armv6 memory management.

-- 
                                                 Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> Wolfgang Grandegger wrote:
>> Hello,
>>
>> I realized that accessing memory allocated with rt_heap_alloc() causes
>> mode switches on ARM i.mx31. The attached patch provides a demo program
>> to demonstrate the problem, which actually does *not* show up on my
>> PowerPC TQM5200 board.
> 
> Ok. I ran your test on an ARMv5 using an old version of Xenomai, and I
> only get a fault on first write to the heap, being either cached or non
> cached.
> 
> I will now run the tests on AT91RM9200 with a more recent version of
> xenomai to confirm that we get the same effect, but if we do, this means
> that we are facing an armv6 specific issue and that for a workaround, we
> need to understand armv6 memory management.

The problem seems to be more general. I have now use the TLSF allocator
from Xenomai-solo for dynamic memory allocation and I still see the mode
switch counter increasing. Again, no problems on my TQM5200 PowerPC
board. I have the impression that the problems are related to armv6.

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>>> Hello,
>>>
>>> I realized that accessing memory allocated with rt_heap_alloc() causes
>>> mode switches on ARM i.mx31. The attached patch provides a demo program
>>> to demonstrate the problem, which actually does *not* show up on my
>>> PowerPC TQM5200 board.
>> Ok. I ran your test on an ARMv5 using an old version of Xenomai, and I
>> only get a fault on first write to the heap, being either cached or non
>> cached.
>>
>> I will now run the tests on AT91RM9200 with a more recent version of
>> xenomai to confirm that we get the same effect, but if we do, this means
>> that we are facing an armv6 specific issue and that for a workaround, we
>> need to understand armv6 memory management.
> 
> The problem seems to be more general. I have now use the TLSF allocator
> from Xenomai-solo for dynamic memory allocation and I still see the mode
> switch counter increasing. Again, no problems on my TQM5200 PowerPC
> board. I have the impression that the problems are related to armv6.

That is because you do unaligned accesses, on ARM they generate faults.

-- 
                                                 Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> Wolfgang Grandegger wrote:
>> Gilles Chanteperdrix wrote:
>>> Wolfgang Grandegger wrote:
>>>> Hello,
>>>>
>>>> I realized that accessing memory allocated with rt_heap_alloc() causes
>>>> mode switches on ARM i.mx31. The attached patch provides a demo program
>>>> to demonstrate the problem, which actually does *not* show up on my
>>>> PowerPC TQM5200 board.
>>> Ok. I ran your test on an ARMv5 using an old version of Xenomai, and I
>>> only get a fault on first write to the heap, being either cached or non
>>> cached.
>>>
>>> I will now run the tests on AT91RM9200 with a more recent version of
>>> xenomai to confirm that we get the same effect, but if we do, this means
>>> that we are facing an armv6 specific issue and that for a workaround, we
>>> need to understand armv6 memory management.
>> The problem seems to be more general. I have now use the TLSF allocator
>> from Xenomai-solo for dynamic memory allocation and I still see the mode
>> switch counter increasing. Again, no problems on my TQM5200 PowerPC
>> board. I have the impression that the problems are related to armv6.
> 
> That is because you do unaligned accesses, on ARM they generate faults.

Good catch. That also explains why a mode switch does not always happen.
I will do more tests tomorrow.

Thanks,

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Hello,
> 
> I realized that accessing memory allocated with rt_heap_alloc() causes
> mode switches on ARM i.mx31. The attached patch provides a demo program
> to demonstrate the problem, which actually does *not* show up on my
> PowerPC TQM5200 board.

Index: xenomai-2.4.x/examples/native/rtheap.c
===================================================================
--- /dev/null
+++ xenomai-2.4.x/examples/native/rtheap.c
@@ -0,0 +1,164 @@
+#include <stdio.h>
+#include <signal.h>
+#include <unistd.h>
+#include <sys/mman.h>
+
+#include <native/task.h>
+#include <native/timer.h>
+#include <native/heap.h>
+
+#define USE_HEAP
+//#define USE_SIGXCPU
+
+#define HEAP_SIZE (1024*1024)
+#define HEAP_MODE 0		/* Local heap. */
+
+RT_HEAP heap_desc;
+RT_TASK demo_task;
+
+
+static int block_sizes[] = {16, 20, 90, 150, 310, 800, 1000};
+
+/* NOTE: error handling omitted. */
+
+void demo(void *arg)
+{
+	RTIME now, previous;
+	void *block;
+	int sizes = sizeof(block_sizes) / sizeof(int);
+	int count = 0;
+	int size, err;
+	int *ptr;
+
+#ifdef USE_SIGXCPU
+	/* Ask Xenomai to warn us upon switches to secondary mode. */
+	rt_task_set_mode(0, T_WARNSW, NULL);
+#endif
+
+	/*
+	 * Arguments: &task (NULL=self),
+	 *            start time,
+	 *            period (here: 1 s)
+	 */
+	rt_task_set_periodic(NULL, TM_NOW, 10000000);
+	previous = rt_timer_read();
+
+	while (1) {
+		rt_task_wait_period(NULL);
+		now = rt_timer_read();
+
+		size = block_sizes[count % sizes];
+#ifdef USE_HEAP
+		/*
+		 * Request a 16-bytes block, asking for a non-blocking call
+		 * since only Xenomai tasks may block.
+		 */
+		err = rt_heap_alloc(&heap_desc, size, TM_NONBLOCK, &block);
+		if (err) {
+			printf("rt_heap_alloc() failed with %d\n", err);
+			break;
+		}
+
+		ptr = (int *)block;
+		*ptr = 0xdeadbeef;
+		if (*ptr != 0xdeadbeef) {
+			printf("Write/read test to heap failed\n");
+			break;
+		}
+
+		ptr = (int *)(block + size - sizeof(int));

This is not guaranteed to be aligned on an int. Could you try:

		ptr = (int *)(((unsigned long) block + size - sizeof(int)) & ~3);

-- 
                                                 Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> Wolfgang Grandegger wrote:
>> Hello,
>>
>> I realized that accessing memory allocated with rt_heap_alloc() causes
>> mode switches on ARM i.mx31. The attached patch provides a demo program
>> to demonstrate the problem, which actually does *not* show up on my
>> PowerPC TQM5200 board.
> 
> Index: xenomai-2.4.x/examples/native/rtheap.c
> ===================================================================
> --- /dev/null
> +++ xenomai-2.4.x/examples/native/rtheap.c
> @@ -0,0 +1,164 @@
> +#include <stdio.h>
> +#include <signal.h>
> +#include <unistd.h>
> +#include <sys/mman.h>
> +
> +#include <native/task.h>
> +#include <native/timer.h>
> +#include <native/heap.h>
> +
> +#define USE_HEAP
> +//#define USE_SIGXCPU
> +
> +#define HEAP_SIZE (1024*1024)
> +#define HEAP_MODE 0		/* Local heap. */
> +
> +RT_HEAP heap_desc;
> +RT_TASK demo_task;
> +
> +
> +static int block_sizes[] = {16, 20, 90, 150, 310, 800, 1000};
> +
> +/* NOTE: error handling omitted. */
> +
> +void demo(void *arg)
> +{
> +	RTIME now, previous;
> +	void *block;
> +	int sizes = sizeof(block_sizes) / sizeof(int);
> +	int count = 0;
> +	int size, err;
> +	int *ptr;
> +
> +#ifdef USE_SIGXCPU
> +	/* Ask Xenomai to warn us upon switches to secondary mode. */
> +	rt_task_set_mode(0, T_WARNSW, NULL);
> +#endif
> +
> +	/*
> +	 * Arguments: &task (NULL=self),
> +	 *            start time,
> +	 *            period (here: 1 s)
> +	 */
> +	rt_task_set_periodic(NULL, TM_NOW, 10000000);
> +	previous = rt_timer_read();
> +
> +	while (1) {
> +		rt_task_wait_period(NULL);
> +		now = rt_timer_read();
> +
> +		size = block_sizes[count % sizes];
> +#ifdef USE_HEAP
> +		/*
> +		 * Request a 16-bytes block, asking for a non-blocking call
> +		 * since only Xenomai tasks may block.
> +		 */
> +		err = rt_heap_alloc(&heap_desc, size, TM_NONBLOCK, &block);
> +		if (err) {
> +			printf("rt_heap_alloc() failed with %d\n", err);
> +			break;
> +		}
> +
> +		ptr = (int *)block;
> +		*ptr = 0xdeadbeef;
> +		if (*ptr != 0xdeadbeef) {
> +			printf("Write/read test to heap failed\n");
> +			break;
> +		}
> +
> +		ptr = (int *)(block + size - sizeof(int));
> 
> This is not guaranteed to be aligned on an int. Could you try:
> 
> 		ptr = (int *)(((unsigned long) block + size - sizeof(int)) & ~3);

With aligned accesses rtheap reports just3 mode switches in both modes,
0 and H_NONCACHED. But already one mode switch is too much!

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>>> Hello,
>>>
>>> I realized that accessing memory allocated with rt_heap_alloc() causes
>>> mode switches on ARM i.mx31. The attached patch provides a demo program
>>> to demonstrate the problem, which actually does *not* show up on my
>>> PowerPC TQM5200 board.
>> Index: xenomai-2.4.x/examples/native/rtheap.c
>> ===================================================================
>> --- /dev/null
>> +++ xenomai-2.4.x/examples/native/rtheap.c
>> @@ -0,0 +1,164 @@
>> +#include <stdio.h>
>> +#include <signal.h>
>> +#include <unistd.h>
>> +#include <sys/mman.h>
>> +
>> +#include <native/task.h>
>> +#include <native/timer.h>
>> +#include <native/heap.h>
>> +
>> +#define USE_HEAP
>> +//#define USE_SIGXCPU
>> +
>> +#define HEAP_SIZE (1024*1024)
>> +#define HEAP_MODE 0		/* Local heap. */
>> +
>> +RT_HEAP heap_desc;
>> +RT_TASK demo_task;
>> +
>> +
>> +static int block_sizes[] = {16, 20, 90, 150, 310, 800, 1000};
>> +
>> +/* NOTE: error handling omitted. */
>> +
>> +void demo(void *arg)
>> +{
>> +	RTIME now, previous;
>> +	void *block;
>> +	int sizes = sizeof(block_sizes) / sizeof(int);
>> +	int count = 0;
>> +	int size, err;
>> +	int *ptr;
>> +
>> +#ifdef USE_SIGXCPU
>> +	/* Ask Xenomai to warn us upon switches to secondary mode. */
>> +	rt_task_set_mode(0, T_WARNSW, NULL);
>> +#endif
>> +
>> +	/*
>> +	 * Arguments: &task (NULL=self),
>> +	 *            start time,
>> +	 *            period (here: 1 s)
>> +	 */
>> +	rt_task_set_periodic(NULL, TM_NOW, 10000000);
>> +	previous = rt_timer_read();
>> +
>> +	while (1) {
>> +		rt_task_wait_period(NULL);
>> +		now = rt_timer_read();
>> +
>> +		size = block_sizes[count % sizes];
>> +#ifdef USE_HEAP
>> +		/*
>> +		 * Request a 16-bytes block, asking for a non-blocking call
>> +		 * since only Xenomai tasks may block.
>> +		 */
>> +		err = rt_heap_alloc(&heap_desc, size, TM_NONBLOCK, &block);
>> +		if (err) {
>> +			printf("rt_heap_alloc() failed with %d\n", err);
>> +			break;
>> +		}
>> +
>> +		ptr = (int *)block;
>> +		*ptr = 0xdeadbeef;
>> +		if (*ptr != 0xdeadbeef) {
>> +			printf("Write/read test to heap failed\n");
>> +			break;
>> +		}
>> +
>> +		ptr = (int *)(block + size - sizeof(int));
>>
>> This is not guaranteed to be aligned on an int. Could you try:
>>
>> 		ptr = (int *)(((unsigned long) block + size - sizeof(int)) & ~3);
> 
> With aligned accesses rtheap reports just3 mode switches in both modes,
> 0 and H_NONCACHED. But already one mode switch is too much!

I hope we get several mode switches because the allocator returns
different pages. To workaround this, we will have to cause a fault on
each page.  We can do this either in kernel or user-space.

-- 
					    Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> Wolfgang Grandegger wrote:
>> Gilles Chanteperdrix wrote:
>>> Wolfgang Grandegger wrote:
>>>> Hello,
>>>>
>>>> I realized that accessing memory allocated with rt_heap_alloc() causes
>>>> mode switches on ARM i.mx31. The attached patch provides a demo program
>>>> to demonstrate the problem, which actually does *not* show up on my
>>>> PowerPC TQM5200 board.
>>> Index: xenomai-2.4.x/examples/native/rtheap.c
>>> ===================================================================
>>> --- /dev/null
>>> +++ xenomai-2.4.x/examples/native/rtheap.c
>>> @@ -0,0 +1,164 @@
>>> +#include <stdio.h>
>>> +#include <signal.h>
>>> +#include <unistd.h>
>>> +#include <sys/mman.h>
>>> +
>>> +#include <native/task.h>
>>> +#include <native/timer.h>
>>> +#include <native/heap.h>
>>> +
>>> +#define USE_HEAP
>>> +//#define USE_SIGXCPU
>>> +
>>> +#define HEAP_SIZE (1024*1024)
>>> +#define HEAP_MODE 0		/* Local heap. */
>>> +
>>> +RT_HEAP heap_desc;
>>> +RT_TASK demo_task;
>>> +
>>> +
>>> +static int block_sizes[] = {16, 20, 90, 150, 310, 800, 1000};
>>> +
>>> +/* NOTE: error handling omitted. */
>>> +
>>> +void demo(void *arg)
>>> +{
>>> +	RTIME now, previous;
>>> +	void *block;
>>> +	int sizes = sizeof(block_sizes) / sizeof(int);
>>> +	int count = 0;
>>> +	int size, err;
>>> +	int *ptr;
>>> +
>>> +#ifdef USE_SIGXCPU
>>> +	/* Ask Xenomai to warn us upon switches to secondary mode. */
>>> +	rt_task_set_mode(0, T_WARNSW, NULL);
>>> +#endif
>>> +
>>> +	/*
>>> +	 * Arguments: &task (NULL=self),
>>> +	 *            start time,
>>> +	 *            period (here: 1 s)
>>> +	 */
>>> +	rt_task_set_periodic(NULL, TM_NOW, 10000000);
>>> +	previous = rt_timer_read();
>>> +
>>> +	while (1) {
>>> +		rt_task_wait_period(NULL);
>>> +		now = rt_timer_read();
>>> +
>>> +		size = block_sizes[count % sizes];
>>> +#ifdef USE_HEAP
>>> +		/*
>>> +		 * Request a 16-bytes block, asking for a non-blocking call
>>> +		 * since only Xenomai tasks may block.
>>> +		 */
>>> +		err = rt_heap_alloc(&heap_desc, size, TM_NONBLOCK, &block);
>>> +		if (err) {
>>> +			printf("rt_heap_alloc() failed with %d\n", err);
>>> +			break;
>>> +		}
>>> +
>>> +		ptr = (int *)block;
>>> +		*ptr = 0xdeadbeef;
>>> +		if (*ptr != 0xdeadbeef) {
>>> +			printf("Write/read test to heap failed\n");
>>> +			break;
>>> +		}
>>> +
>>> +		ptr = (int *)(block + size - sizeof(int));
>>>
>>> This is not guaranteed to be aligned on an int. Could you try:
>>>
>>> 		ptr = (int *)(((unsigned long) block + size - sizeof(int)) & ~3);
>> With aligned accesses rtheap reports just3 mode switches in both modes,
>> 0 and H_NONCACHED. But already one mode switch is too much!
> 
> I hope we get several mode switches because the allocator returns
> different pages. To workaround this, we will have to cause a fault on
> each page.  We can do this either in kernel or user-space.

Doing it in kernel-space, e.g. in xnheap_init_mapped(), does not help.
Anyhow, I think the memory pages get touched in kernel space already here:

http://www.rts.uni-hannover.de/xenomai/lxr/source/ksrc/nucleus/heap.c#096

But the mode switches disappear when I touch the allocated memory (read
and write back) in __map_heap_memory() write after the __real_mmap(:

http://www.rts.uni-hannover.de/xenomai/lxr/source/src/skins/native/heap.c#031

Wolfgang.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Wolfgang Grandegger wrote:
>>>>> Hello,
>>>>>
>>>>> I realized that accessing memory allocated with rt_heap_alloc() causes
>>>>> mode switches on ARM i.mx31. The attached patch provides a demo program
>>>>> to demonstrate the problem, which actually does *not* show up on my
>>>>> PowerPC TQM5200 board.
>>>> Index: xenomai-2.4.x/examples/native/rtheap.c
>>>> ===================================================================
>>>> --- /dev/null
>>>> +++ xenomai-2.4.x/examples/native/rtheap.c
>>>> @@ -0,0 +1,164 @@
>>>> +#include <stdio.h>
>>>> +#include <signal.h>
>>>> +#include <unistd.h>
>>>> +#include <sys/mman.h>
>>>> +
>>>> +#include <native/task.h>
>>>> +#include <native/timer.h>
>>>> +#include <native/heap.h>
>>>> +
>>>> +#define USE_HEAP
>>>> +//#define USE_SIGXCPU
>>>> +
>>>> +#define HEAP_SIZE (1024*1024)
>>>> +#define HEAP_MODE 0		/* Local heap. */
>>>> +
>>>> +RT_HEAP heap_desc;
>>>> +RT_TASK demo_task;
>>>> +
>>>> +
>>>> +static int block_sizes[] = {16, 20, 90, 150, 310, 800, 1000};
>>>> +
>>>> +/* NOTE: error handling omitted. */
>>>> +
>>>> +void demo(void *arg)
>>>> +{
>>>> +	RTIME now, previous;
>>>> +	void *block;
>>>> +	int sizes = sizeof(block_sizes) / sizeof(int);
>>>> +	int count = 0;
>>>> +	int size, err;
>>>> +	int *ptr;
>>>> +
>>>> +#ifdef USE_SIGXCPU
>>>> +	/* Ask Xenomai to warn us upon switches to secondary mode. */
>>>> +	rt_task_set_mode(0, T_WARNSW, NULL);
>>>> +#endif
>>>> +
>>>> +	/*
>>>> +	 * Arguments: &task (NULL=self),
>>>> +	 *            start time,
>>>> +	 *            period (here: 1 s)
>>>> +	 */
>>>> +	rt_task_set_periodic(NULL, TM_NOW, 10000000);
>>>> +	previous = rt_timer_read();
>>>> +
>>>> +	while (1) {
>>>> +		rt_task_wait_period(NULL);
>>>> +		now = rt_timer_read();
>>>> +
>>>> +		size = block_sizes[count % sizes];
>>>> +#ifdef USE_HEAP
>>>> +		/*
>>>> +		 * Request a 16-bytes block, asking for a non-blocking call
>>>> +		 * since only Xenomai tasks may block.
>>>> +		 */
>>>> +		err = rt_heap_alloc(&heap_desc, size, TM_NONBLOCK, &block);
>>>> +		if (err) {
>>>> +			printf("rt_heap_alloc() failed with %d\n", err);
>>>> +			break;
>>>> +		}
>>>> +
>>>> +		ptr = (int *)block;
>>>> +		*ptr = 0xdeadbeef;
>>>> +		if (*ptr != 0xdeadbeef) {
>>>> +			printf("Write/read test to heap failed\n");
>>>> +			break;
>>>> +		}
>>>> +
>>>> +		ptr = (int *)(block + size - sizeof(int));
>>>>
>>>> This is not guaranteed to be aligned on an int. Could you try:
>>>>
>>>> 		ptr = (int *)(((unsigned long) block + size - sizeof(int)) & ~3);
>>> With aligned accesses rtheap reports just3 mode switches in both modes,
>>> 0 and H_NONCACHED. But already one mode switch is too much!
>> I hope we get several mode switches because the allocator returns
>> different pages. To workaround this, we will have to cause a fault on
>> each page.  We can do this either in kernel or user-space.

I checked this, and this is true: we get a fault by different page
returned by the allocator.

> 
> Doing it in kernel-space, e.g. in xnheap_init_mapped(), does not help.
> Anyhow, I think the memory pages get touched in kernel space already here:

That is because you have to fault the user-space addresses, not the
kernel-space addresses, you have to use something like handle_mm_fault.

> 
> http://www.rts.uni-hannover.de/xenomai/lxr/source/ksrc/nucleus/heap.c#096
> 
> But the mode switches disappear when I touch the allocated memory (read
> and write back) in __map_heap_memory() write after the __real_mmap(:
> 
> http://www.rts.uni-hannover.de/xenomai/lxr/source/src/skins/native/heap.c#031

The problem if we do it in user-space is that we would have to do this
for every uses of the heaps, that is native skin heaps, posix skin
heaps, mutexes heaps, and every rtdm driver using shared heaps too.

-- 
                                                 Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>> +		ptr = (int *)(block + size - sizeof(int));
>>
>> This is not guaranteed to be aligned on an int. Could you try:
>>
>> 		ptr = (int *)(((unsigned long) block + size - sizeof(int)) & ~3);
> 
> With aligned accesses rtheap reports just3 mode switches in both modes,
> 0 and H_NONCACHED. But already one mode switch is too much!

This should be solved by the current versions of branch v2.4.x and trunk.

-- 
                                                 Gilles.

Re: [Xenomai-help] Mode switch when using RT heap on ARM

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Gilles Chanteperdrix wrote:
>> Wolfgang Grandegger wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Wolfgang Grandegger wrote:
>>>>> Gilles Chanteperdrix wrote:
>>>>>> Wolfgang Grandegger wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I realized that accessing memory allocated with rt_heap_alloc()
causes
>>>>>>> mode switches on ARM i.mx31. The attached patch provides a demo
program
>>>>>>> to demonstrate the problem, which actually does *not* show up on
my PowerPC TQM5200 board.
>>>>>> On ARM, we normally map the heaps uncacheable, this should not be
necessary on ARMv6, but I am afraid we then get the fault on first
access.
>>>>> Is each cache-line of the heap not touched automatically when the heap
>>>>> gets created? I thought it's necessary for other archs as well.
>>>> No, the thing which comes near to this is the workaround in I-pipe of
the
>>>> way pages are write-protected upon fork. But this happens only upon
fork.
>>>> I am not sure I understand all the subtleties of ARM memory
>>>> management,
>>>> but I think this fault on first write is the way the "dirty" bit is
implemented.
>>> Well, it seems not to be that simple. My attached rtheap example program
>>> behaves the following way:
>>>
>>> - Heap-mode=0: I see plenty of mode switches also for read-only:
>>
>> Maybe that is expected on ARM ? But what I do not understand is that
you get 300 modes switches where as your example makes two faults every
10ms during 10 seconds, so, you should see something like 1000 modes
switches.
>
> And even more because the program writes/reads at the beginning and end
of the buffer. I realized that as well. Not every read/write seems to
provoke a mode switch.

Maybe because when the heap returns the same page, if the fault really
happens on first write, you do not see the fault again.

>
>>>   # cat /proc/xenomai/stat ;sleep 10; cat /proc/xenomai/stat
>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>     0  0      0          2951       0     00400080   99.7  ROOT 0  929
   102        398        3     00300184    0.0  rtheap 0  0      0
         23005      0     00000000    0.1  IRQ29:
>>> [timer]
>>>   CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>>     0  0      0          4287       0     00400080   99.4  ROOT 0  929
   435        1734       3     00300184    0.4  rtheap 0  0      0
         25010      0     00000000    0.1  IRQ29:
>>> [timer]
>>>
>>> - Heap-mode=H_NONCACHED: I see just 2 mode switches but the system
>>>   gets very slow:
>>
>> That is expected. But where do you get the 2 mode switches ?
>
> When the task writes to the heap memory for the first time. And I just
realized that the write/read to the end of the buffer makes real
trouble. The task seems to hang (or wait for something):
>
> -bash-3.2# cat sched
> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>   0  0       99      0          0          master    R          ROOT 0 
1114    99      100000000  0          master    X          rtheap
>
> At the same time the system gets slow.
>
> So far I understood from your comments that rtheap is not really usable
on ARM, right? What other option do I have?

No, I would have expected heap to be usable when non cachable, but to be
slow, that is just what I said.

-- 
                    Gilles.



-- 
                    Gilles.