[Xenomai] rt_heap_alloc priority inversion

[Xenomai] rt_heap_alloc priority inversion

[George Broz]

Hello All,

I'm running Linux 2.6.37.6 w/Xenomai 2.6.1, native API 
on x86 (Atom, SMP, 32-bit). 

I have two tasks, both running in Xenomai user-space. 
One is priority=99 and blocks on rt_intr_wait(), 
running every 250 us with 125us margin. This task does 
not make direct calls to rt_heap_alloc/free().

The other task is priority=50, wakes 1msec, sleeps 1msec.
This task routinely makes rt_heap_alloc/free calls. 

When the priority=50 task makes rt_heap_alloc() calls
for "size" greater than 5 MB, the priority=99 task
appears to be held off from executing for several ms.

Plenty of memory is available (vmalloc=320M as linux 
boot arg). TM_NONBLOCK is used when calling 
rt_heap_alloc(). Both "0" and H_PRIO have been tried
for "mode" when creating the area with rt_heap_create().

Is this expected behavior? 

How can large memory allocations be built without a 
"realloc" equivalent offered in the memory heap 
services?


Thanks in advance!
---
George Broz
Moog, Inc. Industrial Group

Re: [Xenomai] rt_heap_alloc priority inversion

[Gilles Chanteperdrix]

On 10/25/2013 11:59 PM, George Broz wrote:
> Hello All,
> 
> I'm running Linux 2.6.37.6 w/Xenomai 2.6.1, native API 
> on x86 (Atom, SMP, 32-bit). 

Do you observe the same issues with Xenomai latest release?


-- 
                                                                Gilles.

Re: [Xenomai] rt_heap_alloc priority inversion

[George Broz]

-----Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote: -----

>On 10/25/2013 11:59 PM, George Broz wrote:
>> Hello All,
>> 
>> I'm running Linux 2.6.37.6 w/Xenomai 2.6.1, native API 
>> on x86 (Atom, SMP, 32-bit). 
>
>Do you observe the same issues with Xenomai latest release?
>

I can't easily change kernels. I would not be in a position 
to even try 2.6.3 for a couple of weeks. If this behavior
is not expected, and 2.6.3 may potentially provide a fix,
then I would have to make a proposal. Any insights 
appreciated.


Thanks,
--George

Re: [Xenomai] rt_heap_alloc priority inversion

[Philippe Gerum]

On 10/25/2013 11:59 PM, George Broz wrote:
> Hello All,
>
> I'm running Linux 2.6.37.6 w/Xenomai 2.6.1, native API
> on x86 (Atom, SMP, 32-bit).
>
> I have two tasks, both running in Xenomai user-space.
> One is priority=99 and blocks on rt_intr_wait(),
> running every 250 us with 125us margin. This task does
> not make direct calls to rt_heap_alloc/free().
>
> The other task is priority=50, wakes 1msec, sleeps 1msec.
> This task routinely makes rt_heap_alloc/free calls.
>
> When the priority=50 task makes rt_heap_alloc() calls
> for "size" greater than 5 MB, the priority=99 task
> appears to be held off from executing for several ms.
>
> Plenty of memory is available (vmalloc=320M as linux
> boot arg).

The size that matters in how much memory you assigned to the heap in 
rt_heap_create().

  TM_NONBLOCK is used when calling
> rt_heap_alloc(). Both "0" and H_PRIO have been tried
> for "mode" when creating the area with rt_heap_create().
>
> Is this expected behavior?
>
> How can large memory allocations be built without a
> "realloc" equivalent offered in the memory heap
> services?
>
>

There is no reallocation in the rt heap services, ever. If H_SINGLE was 
passed to rt_heap_create(), there is not even any "allocation", the 
address of the heap memory is returned in that case, as it is viewed as 
a single buffer.

Assuming you don't pass H_SINGLE when creating the heap, the only reason 
to get delays in non-blocking allocation mode would be that a minor VM 
fault unexpectedly happens when the 5MB block is being pulled from the 
heap, when writing our meta-data to that memory area. That would mean 
that your calling thread switches to secondary mode for handling the 
fault, and gets delayed for this reason.

You should first try checking this assumption, by enabling T_WARNSW for 
your thread. If this is confirmed, then there could be something wrong 
in the i-pipe layer for your kernel release.

-- 
Philippe.

Re: [Xenomai] rt_heap_alloc priority inversion

[George Broz]

-----Philippe Gerum <rpm@xenomai.org> wrote: -----
>
>On 10/25/2013 11:59 PM, George Broz wrote:
>> Hello All,
>>
>> I'm running Linux 2.6.37.6 w/Xenomai 2.6.1, native API
>> on x86 (Atom, SMP, 32-bit).
>>
>> I have two tasks, both running in Xenomai user-space.
>> One is priority=99 and blocks on rt_intr_wait(),
>> running every 250 us with 125us margin. This task does
>> not make direct calls to rt_heap_alloc/free().
>>
>> The other task is priority=50, wakes 1msec, sleeps 1msec.
>> This task routinely makes rt_heap_alloc/free calls.
>>
>> When the priority=50 task makes rt_heap_alloc() calls
>> for "size" greater than 5 MB, the priority=99 task
>> appears to be held off from executing for several ms.
>>
>> Plenty of memory is available (vmalloc=320M as linux
>> boot arg).
>
>The size that matters in how much memory you assigned to the heap in
>
>rt_heap_create().

Yes, sorry. Passing 256 MB (as 268,435,456) to rt_heap_create().

>
>  TM_NONBLOCK is used when calling
>> rt_heap_alloc(). Both "0" and H_PRIO have been tried
>> for "mode" when creating the area with rt_heap_create().
>>
>> Is this expected behavior?
>>
>> How can large memory allocations be built without a
>> "realloc" equivalent offered in the memory heap
>> services?
>>
>>
>
>There is no reallocation in the rt heap services, ever. If H_SINGLE
>was 
>passed to rt_heap_create(), there is not even any "allocation", the 
>address of the heap memory is returned in that case, as it is viewed
>as 
>a single buffer.
>
>Assuming you don't pass H_SINGLE when creating the heap, 


We're not passing H_SINGLE when creating the heap. The only mode flag 
passed to _create() is H_PRIO or H_FIFO ("0") - have tried both, 
and resulting behavior is the same.


>the only reason 
>to get delays in non-blocking allocation mode would be that a minor
>VM 
>fault unexpectedly happens when the 5MB block is being pulled from
>the 
>heap, when writing our meta-data to that memory area. That would mean
>
>that your calling thread switches to secondary mode for handling the
>
>fault, and gets delayed for this reason.
>
>You should first try checking this assumption, by enabling T_WARNSW
>for 
>your thread. If this is confirmed, then there could be something
>wrong 
>in the i-pipe layer for your kernel release.

We always run with T_WARNSW turned on for the Xenomai tasks that
should never switch to secondary. A backtrace is generated if 
a secondary transition occurs. We have used this mechanism 
extensively in our development. When we attempt to allocate 
5 MB, neither the calling task (prio=50), nor the high priority
task (prio=99) that gets held off logs a secondary mode 
transition.


Going a step further, we can run in "simulation" mode where the
hardware interrupt source (on which rt_intr_wait() is waiting) 
is replaced with rt_task_set_periodic()/rt_task_wait_period()
and the same behavior results.

Any ideas?

>
>-- 
>Philippe.


Thanks for your help! Much appreciated!
--George

Re: [Xenomai] rt_heap_alloc priority inversion

[Gilles Chanteperdrix]

On 10/27/2013 05:54 PM, George Broz wrote:
> -----Philippe Gerum <rpm@xenomai.org> wrote: -----
>>
>> On 10/25/2013 11:59 PM, George Broz wrote:
>>> Hello All,
>>>
>>> I'm running Linux 2.6.37.6 w/Xenomai 2.6.1, native API
>>> on x86 (Atom, SMP, 32-bit).
>>>
>>> I have two tasks, both running in Xenomai user-space.
>>> One is priority=99 and blocks on rt_intr_wait(),
>>> running every 250 us with 125us margin. This task does
>>> not make direct calls to rt_heap_alloc/free().
>>>
>>> The other task is priority=50, wakes 1msec, sleeps 1msec.
>>> This task routinely makes rt_heap_alloc/free calls.
>>>
>>> When the priority=50 task makes rt_heap_alloc() calls
>>> for "size" greater than 5 MB, the priority=99 task
>>> appears to be held off from executing for several ms.
>>>
>>> Plenty of memory is available (vmalloc=320M as linux
>>> boot arg).
>>
>> The size that matters in how much memory you assigned to the heap in
>>
>> rt_heap_create().
> 
> Yes, sorry. Passing 256 MB (as 268,435,456) to rt_heap_create().
> 
>>
>>  TM_NONBLOCK is used when calling
>>> rt_heap_alloc(). Both "0" and H_PRIO have been tried
>>> for "mode" when creating the area with rt_heap_create().
>>>
>>> Is this expected behavior?
>>>
>>> How can large memory allocations be built without a
>>> "realloc" equivalent offered in the memory heap
>>> services?
>>>
>>>
>>
>> There is no reallocation in the rt heap services, ever. If H_SINGLE
>> was 
>> passed to rt_heap_create(), there is not even any "allocation", the 
>> address of the heap memory is returned in that case, as it is viewed
>> as 
>> a single buffer.
>>
>> Assuming you don't pass H_SINGLE when creating the heap, 
> 
> 
> We're not passing H_SINGLE when creating the heap. The only mode flag 
> passed to _create() is H_PRIO or H_FIFO ("0") - have tried both, 
> and resulting behavior is the same.
> 
> 
>> the only reason 
>> to get delays in non-blocking allocation mode would be that a minor
>> VM 
>> fault unexpectedly happens when the 5MB block is being pulled from
>> the 
>> heap, when writing our meta-data to that memory area. That would mean
>>
>> that your calling thread switches to secondary mode for handling the
>>
>> fault, and gets delayed for this reason.
>>
>> You should first try checking this assumption, by enabling T_WARNSW
>> for 
>> your thread. If this is confirmed, then there could be something
>> wrong 
>> in the i-pipe layer for your kernel release.
> 
> We always run with T_WARNSW turned on for the Xenomai tasks that
> should never switch to secondary. A backtrace is generated if 
> a secondary transition occurs. We have used this mechanism 
> extensively in our development. When we attempt to allocate 
> 5 MB, neither the calling task (prio=50), nor the high priority
> task (prio=99) that gets held off logs a secondary mode 
> transition.
> 
> 
> Going a step further, we can run in "simulation" mode where the
> hardware interrupt source (on which rt_intr_wait() is waiting) 
> is replaced with rt_task_set_periodic()/rt_task_wait_period()
> and the same behavior results.
> 
> Any ideas?

I would suggest again trying the latest version, and it does not mean
updating the kernel.
Another idea would be to run the test without CONFIG_SMP, or booting
with nr_cpus=1 if you are using hyper threading.


-- 
                                                                Gilles.

Re: [Xenomai] rt_heap_alloc priority inversion

[Gilles Chanteperdrix]

On 10/27/2013 05:54 PM, George Broz wrote:
> -----Philippe Gerum <rpm@xenomai.org> wrote: -----
>>
>> On 10/25/2013 11:59 PM, George Broz wrote:
>>> Hello All,
>>>
>>> I'm running Linux 2.6.37.6 w/Xenomai 2.6.1, native API
>>> on x86 (Atom, SMP, 32-bit).
>>>
>>> I have two tasks, both running in Xenomai user-space.
>>> One is priority=99 and blocks on rt_intr_wait(),
>>> running every 250 us with 125us margin. This task does
>>> not make direct calls to rt_heap_alloc/free().
>>>
>>> The other task is priority=50, wakes 1msec, sleeps 1msec.
>>> This task routinely makes rt_heap_alloc/free calls.
>>>
>>> When the priority=50 task makes rt_heap_alloc() calls
>>> for "size" greater than 5 MB, the priority=99 task
>>> appears to be held off from executing for several ms.
>>>
>>> Plenty of memory is available (vmalloc=320M as linux
>>> boot arg).
>>
>> The size that matters in how much memory you assigned to the heap in
>>
>> rt_heap_create().
> 
> Yes, sorry. Passing 256 MB (as 268,435,456) to rt_heap_create().

I believe the rt_heap is not meant for such large sizes: to satisfy a 5
MB allocation using 4096 bytes pages, it has to allocate 1280 pages,
that means a lot of housekeeping which happens with a spinlock locked,
irqs off. So, for your case you would want to use a larger page size,
say 1MB. Please try the following patch which is not a fix, but simply
an attempt at finding whether the page size is what causes the issue.

diff --git a/ksrc/nucleus/heap.c b/ksrc/nucleus/heap.c
index 568ba27..3a9105a 100644
--- a/ksrc/nucleus/heap.c
+++ b/ksrc/nucleus/heap.c
@@ -1362,7 +1362,7 @@ int xnheap_init_mapped(xnheap_t *heap, u_long
heapsize, int memflags)
 	if (heapbase == NULL)
 		return -ENOMEM;

-	err = xnheap_init(heap, heapbase, heapsize, PAGE_SIZE);
+	err = xnheap_init(heap, heapbase, heapsize, 1024 * 1024);
 	if (err) {
 		__unreserve_and_free_heap(heapbase, heapsize, memflags);
 		return err;

-- 
                                                                Gilles.