[Xenomai] Performance issue with memory allocators

[Xenomai] Performance issue with memory allocators

[Philippe Gerum]

After a comprehensive series of tests, it is now obvious that
heapobj-pshared has gruesome performances when releasing bursts of
small-sized objects. Allocation of large objects is not particularly
efficient either, although not as bad. The Cobalt core allocator is
affected by the same bug too.

Those new tests are available from the wip/heapmem branch, they are
based on the smokey framework (e.g. smokey --run=memory_* --verbose=2).

Work is ongoing into the wip/heapmem branch to fix the mess, first
addressing the issue with the core (in-kernel) allocator,
heapobj-pshared next. Please note that wip/heapmem is a development
branch I'm working on for testing improvements - hopefully - to the
memory allocators, which should not be used in production.

I would strongly recommend that users running single-process
applications - i.e. not needing the shared heap feature - make sure to
turn off heapobj-pshared when configuring, by passing --disable-pshared
to the configuration script (or leave the feature to the default off state).

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Philippe Gerum]

On 04/27/2018 08:23 PM, Philippe Gerum wrote:
> 
> After a comprehensive series of tests, it is now obvious that
> heapobj-pshared has gruesome performances when releasing bursts of
> small-sized objects. Allocation of large objects is not particularly
> efficient either, although not as bad. The Cobalt core allocator is
> affected by the same bug too.
> 
> Those new tests are available from the wip/heapmem branch, they are
> based on the smokey framework (e.g. smokey --run=memory_* --verbose=2).
> 
> Work is ongoing into the wip/heapmem branch to fix the mess, first
> addressing the issue with the core (in-kernel) allocator,
> heapobj-pshared next. Please note that wip/heapmem is a development
> branch I'm working on for testing improvements - hopefully - to the
> memory allocators, which should not be used in production.
> 
> I would strongly recommend that users running single-process
> applications - i.e. not needing the shared heap feature - make sure to
> turn off heapobj-pshared when configuring, by passing --disable-pshared
> to the configuration script (or leave the feature to the default off state).
> 

The situation is now fixed in the -next branch for the core (in kernel)
heap manager. A set of smokey tests exercising and benchmarking all
memory allocators has been merged too. They may run from seconds to
several minutes depending on the processing horsepower available with
the platform [1]. Reducing the number of allocation rounds for running a
memory pattern check with random block sizes helps in limiting the test
duration. e.g.

~ # smokey --run=memory_coreheap --verbose=2 random_alloc_rounds=16
== memcheck started on coreheap at Thu Jan  1 01:21:50 1970

     seq_heap_size=2048k
     random_alloc_rounds=16
     pattern_heap_size=128k
     pattern_check_rounds=128

heapmem is a new user-space allocator which is currently used as a
drop-in replacement for TLSF in this branch. This is mainly a workhorse
code I used for refining the new allocation algorithm, which is now
implemented by the Cobalt core for managing its memory heaps. It may or
may not be kept, depending on a general consensus about the matter.
Passing --with-localmem=tlsf when configuring reverts this setting,
switching back to TLSF for managing process local memory.

coreheap is the nickname of the in-kernel allocator (aka xnheap).
CONFIG_XENO_DRIVERS_HEAPCHECK must be enabled in the Kconfig to run the
corresponding smokey test.

As the benchmark figures show when running 'memory_pshared', the shared
memory allocator still needs to be fixed. This is next on my list.

Whether those major fixes should be backported to the -stable branch at
some point is an open question. The results obtained from the test and
benchmarking suite mentioned earlier should help in deciding about this
issue.

[1] # smokey --list=memory*
#6   memory_coreheap
	Check for the Cobalt core allocator sanity.
	seq_heap_size=<size[K|M|G]>	max. heap size for sequential alloc tests
	pattern_heap_size=<size[K|M|G]>	max. heap size for pattern check test
	random_alloc_rounds=<N>		# of rounds of random-size allocations
	pattern_check_rounds=<N>	# of rounds of pattern check tests
	max_results=<N>	# of result lines (worst-case first, -1=all)
	Set --verbose=2 for detailed runtime statistics.

#7   memory_heapmem
	Check for the heapmem allocator sanity.
	seq_heap_size=<size[K|M|G]>	max. heap size for sequential alloc tests
	pattern_heap_size=<size[K|M|G]>	max. heap size for pattern check test
	random_alloc_rounds=<N>		# of rounds of random-size allocations
	pattern_check_rounds=<N>	# of rounds of pattern check tests
	max_results=<N>	# of result lines (worst-case first, -1=all)
	Set --verbose=2 for detailed runtime statistics.

#8   memory_tlsf
	Check for the TLSF allocator sanity.
	seq_heap_size=<size[K|M|G]>	max. heap size for sequential alloc tests
	pattern_heap_size=<size[K|M|G]>	max. heap size for pattern check test
	random_alloc_rounds=<N>		# of rounds of random-size allocations
	pattern_check_rounds=<N>	# of rounds of pattern check tests
	max_results=<N>	# of result lines (worst-case first, -1=all)
	Set --verbose=2 for detailed runtime statistics.

#26  memory_pshared
	Check for the pshared allocator sanity.
	seq_heap_size=<size[K|M|G]>	max. heap size for sequential alloc tests
	pattern_heap_size=<size[K|M|G]>	max. heap size for pattern check test
	random_alloc_rounds=<N>		# of rounds of random-size allocations
	pattern_check_rounds=<N>	# of rounds of pattern check tests
	max_results=<N>	# of result lines (worst-case first, -1=all)
	Set --verbose=2 for detailed runtime statistics.

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Philippe Gerum]

On 05/20/2018 08:15 PM, Philippe Gerum wrote:
> On 04/27/2018 08:23 PM, Philippe Gerum wrote:
>>
>> After a comprehensive series of tests, it is now obvious that
>> heapobj-pshared has gruesome performances when releasing bursts of
>> small-sized objects. Allocation of large objects is not particularly
>> efficient either, although not as bad. The Cobalt core allocator is
>> affected by the same bug too.
>>
>> Those new tests are available from the wip/heapmem branch, they are
>> based on the smokey framework (e.g. smokey --run=memory_* --verbose=2).
>>
>> Work is ongoing into the wip/heapmem branch to fix the mess, first
>> addressing the issue with the core (in-kernel) allocator,
>> heapobj-pshared next. Please note that wip/heapmem is a development
>> branch I'm working on for testing improvements - hopefully - to the
>> memory allocators, which should not be used in production.
>>
>> I would strongly recommend that users running single-process
>> applications - i.e. not needing the shared heap feature - make sure to
>> turn off heapobj-pshared when configuring, by passing --disable-pshared
>> to the configuration script (or leave the feature to the default off state).
>>
> > As the benchmark figures show when running 'memory_pshared', the shared
> memory allocator still needs to be fixed. This is next on my list.
The pshared allocator is now fixed in the next branch.

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder]

   On 6/8/2018 7:48 AM, Philippe Gerum wrote:

On 05/20/2018 08:15 PM, Philippe Gerum wrote:

On 04/27/2018 08:23 PM, Philippe Gerum wrote:

After a comprehensive series of tests, it is now obvious that
heapobj-pshared has gruesome performances when releasing bursts of
small-sized objects. Allocation of large objects is not particularly
efficient either, although not as bad. The Cobalt core allocator is
affected by the same bug too.

Those new tests are available from the wip/heapmem branch, they are
based on the smokey framework (e.g. smokey --run=memory_* --verbose=2).

Work is ongoing into the wip/heapmem branch to fix the mess, first
addressing the issue with the core (in-kernel) allocator,
heapobj-pshared next. Please note that wip/heapmem is a development
branch I'm working on for testing improvements - hopefully - to the
memory allocators, which should not be used in production.

I would strongly recommend that users running single-process
applications - i.e. not needing the shared heap feature - make sure to
turn off heapobj-pshared when configuring, by passing --disable-pshared
to the configuration script (or leave the feature to the default off state).

As the benchmark figures show when running 'memory_pshared', the shared

memory allocator still needs to be fixed. This is next on my list.

The pshared allocator is now fixed in the next branch.


   I tried to build with the -next branch, I'm not sure if I'm doing
   something wrong, but I'm getting the errors below.  I thought perhaps
   the "struct shared_heap" should have been "struct shared_heap_memory",
   so I made that change, and that resulted in the second set of errors
   shown below that, then I decided time to post to the group:
   Thanks,
   Steve
   _______________________________________________________________________

   make[3]: Entering directory
   '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
     CC       regd.o
     CC       fs-common.o
   In file included from
   ../../../../../xenomai-3/include/cobalt/wrappers.h:21:0,
                    from
   /home/sdf/xenobuild/imx-xenomai/xenomai-3/include/cobalt/time.h:26,
                    from
   /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/sys/t
   ypes.h:132,
                    from
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:19:
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
   a?~open_heapsa?T:
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10:
   error: dereferencing pointer to incomplete type a?~struct
   shared_heapa?T
      heap = container_of(obj, struct shared_heap, memspec);
             ^
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10:
   error: initialization from incompatible pointer type
   [-Werror=incompatible-pointer-types]
      heap = container_of(obj, struct shared_heap, memspec);
             ^
   In file included from
   /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/strin
   g.h:32:0,
                    from
   ../../../../../xenomai-3/include/copperplate/heapobj.h:24,
                    from
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:27:
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:35:
   error: invalid use of undefined type a?~struct shared_heapa?T
      heap = container_of(obj, struct shared_heap, memspec);
                                      ^
   _______________________________________________________________________

   This is with "struct shared_heap" changed to "struct
   shared_heap_memory":
   make[3]: Entering directory
   '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
     CC       regd.o
     CC       fs-common.o
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
   aopen_heapsa:
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:249:17:
   error: astruct shared_heap_memorya has no member named aubytesa
      p->used = heap->ubytes;
                    ^
   ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:250:18:
   error: astruct shared_heap_memorya has no member named atotala
      p->total = heap->total;
                     ^
   Makefile:472: recipe for target 'fs-common.o' failed
   make[3]: *** [fs-common.o] Error 1

Re: [Xenomai] Performance issue with memory allocators

[Philippe Gerum]

On 06/11/2018 10:37 PM, Steve Freyder wrote:
> On 6/8/2018 7:48 AM, Philippe Gerum wrote:
>> On 05/20/2018 08:15 PM, Philippe Gerum wrote:
>>> On 04/27/2018 08:23 PM, Philippe Gerum wrote:
>>>> After a comprehensive series of tests, it is now obvious that
>>>> heapobj-pshared has gruesome performances when releasing bursts of
>>>> small-sized objects. Allocation of large objects is not particularly
>>>> efficient either, although not as bad. The Cobalt core allocator is
>>>> affected by the same bug too.
>>>>
>>>> Those new tests are available from the wip/heapmem branch, they are
>>>> based on the smokey framework (e.g. smokey --run=memory_* --verbose=2).
>>>>
>>>> Work is ongoing into the wip/heapmem branch to fix the mess, first
>>>> addressing the issue with the core (in-kernel) allocator,
>>>> heapobj-pshared next. Please note that wip/heapmem is a development
>>>> branch I'm working on for testing improvements - hopefully - to the
>>>> memory allocators, which should not be used in production.
>>>>
>>>> I would strongly recommend that users running single-process
>>>> applications - i.e. not needing the shared heap feature - make sure to
>>>> turn off heapobj-pshared when configuring, by passing --disable-pshared
>>>> to the configuration script (or leave the feature to the default off state).
>>>>
>>>> As the benchmark figures show when running 'memory_pshared', the shared
>>> memory allocator still needs to be fixed. This is next on my list.
>> The pshared allocator is now fixed in the next branch.
>>
> I tried to build with the -next branch, I'm not sure if I'm doing
> something wrong, but I'm getting the errors below.  I thought perhaps
> the "struct shared_heap" should have been "struct shared_heap_memory",
> so I made that change, and that resulted in the second set of errors
> shown below that, then I decided time to post to the group:
> 
> Thanks,
> Steve
> 
> ------------------------------------------------------------------------
> 
> make[3]: Entering directory
> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>   CC       regd.o
>   CC       fs-common.o
> In file included from
> ../../../../../xenomai-3/include/cobalt/wrappers.h:21:0,
>                  from
> /home/sdf/xenobuild/imx-xenomai/xenomai-3/include/cobalt/time.h:26,
>                  from
> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/sys/types.h:132,
>                  from
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:19:
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
> â?~open_heapsâ?T:
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
> dereferencing pointer to incomplete type â?~struct shared_heapâ?T
>    heap = container_of(obj, struct shared_heap, memspec);
>           ^
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
> initialization from incompatible pointer type
> [-Werror=incompatible-pointer-types]
>    heap = container_of(obj, struct shared_heap, memspec);
>           ^
> In file included from
> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/string.h:32:0,
>                  from
> ../../../../../xenomai-3/include/copperplate/heapobj.h:24,
>                  from
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:27:
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:35: error:
> invalid use of undefined type â?~struct shared_heapâ?T
>    heap = container_of(obj, struct shared_heap, memspec);
>                                    ^
> ------------------------------------------------------------------------
> This is with "struct shared_heap" changed to "struct shared_heap_memory":
> 
> make[3]: Entering directory
> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>   CC       regd.o
>   CC       fs-common.o
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
> âopen_heapsâ:
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:249:17: error:
> âstruct shared_heap_memoryâ has no member named âubytesâ
>    p->used = heap->ubytes;
>                  ^
> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:250:18: error:
> âstruct shared_heap_memoryâ has no member named âtotalâ
>    p->total = heap->total;
>                   ^
> Makefile:472: recipe for target 'fs-common.o' failed
> make[3]: *** [fs-common.o] Error 1
> 

This may help:

diff --git a/lib/copperplate/regd/fs-common.c b/lib/copperplate/regd/fs-common.c
index ef2e6dae9..e0bdd3d04 100644
--- a/lib/copperplate/regd/fs-common.c
+++ b/lib/copperplate/regd/fs-common.c
@@ -211,8 +211,8 @@ int open_heaps(struct fsobj *fsobj, void *priv)
 {
 	struct sysgroup_memspec *obj, *tmp;
 	struct heap_data *heap_data, *p;
+	struct shared_heap_memory *heap;
 	struct fsobstack *o = priv;
-	struct shared_heap *heap;
 	int ret, count, len = 0;
 
 	ret = heapobj_bind_session(__copperplate_setup_data.session_label);
@@ -244,10 +244,10 @@ int open_heaps(struct fsobj *fsobj, void *priv)
 	for_each_sysgroup(obj, tmp, heap) {
 		if (p - heap_data >= count)
 			break;
-		heap = container_of(obj, struct shared_heap, memspec);
+		heap = container_of(obj, struct shared_heap_memory, memspec);
 		namecpy(p->name, heap->name);
-		p->used = heap->ubytes;
-		p->total = heap->total;
+		p->used = heap->used_size;
+		p->total = heap->usable_size;
 		p++;
 	}

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder]

On 6/12/2018 1:39 AM, Philippe Gerum wrote:
> On 06/11/2018 10:37 PM, Steve Freyder wrote:
>> On 6/8/2018 7:48 AM, Philippe Gerum wrote:
>>> On 05/20/2018 08:15 PM, Philippe Gerum wrote:
>>>> On 04/27/2018 08:23 PM, Philippe Gerum wrote:
>>>>> After a comprehensive series of tests, it is now obvious that
>>>>> heapobj-pshared has gruesome performances when releasing bursts of
>>>>> small-sized objects. Allocation of large objects is not particularly
>>>>> efficient either, although not as bad. The Cobalt core allocator is
>>>>> affected by the same bug too.
>>>>>
>>>>> Those new tests are available from the wip/heapmem branch, they are
>>>>> based on the smokey framework (e.g. smokey --run=memory_* --verbose=2).
>>>>>
>>>>> Work is ongoing into the wip/heapmem branch to fix the mess, first
>>>>> addressing the issue with the core (in-kernel) allocator,
>>>>> heapobj-pshared next. Please note that wip/heapmem is a development
>>>>> branch I'm working on for testing improvements - hopefully - to the
>>>>> memory allocators, which should not be used in production.
>>>>>
>>>>> I would strongly recommend that users running single-process
>>>>> applications - i.e. not needing the shared heap feature - make sure to
>>>>> turn off heapobj-pshared when configuring, by passing --disable-pshared
>>>>> to the configuration script (or leave the feature to the default off state).
>>>>>
>>>>> As the benchmark figures show when running 'memory_pshared', the shared
>>>> memory allocator still needs to be fixed. This is next on my list.
>>> The pshared allocator is now fixed in the next branch.
>>>
>> I tried to build with the -next branch, I'm not sure if I'm doing
>> something wrong, but I'm getting the errors below.  I thought perhaps
>> the "struct shared_heap" should have been "struct shared_heap_memory",
>> so I made that change, and that resulted in the second set of errors
>> shown below that, then I decided time to post to the group:
>>
>> Thanks,
>> Steve
>>
>> ------------------------------------------------------------------------
>>
>> make[3]: Entering directory
>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>    CC       regd.o
>>    CC       fs-common.o
>> In file included from
>> ../../../../../xenomai-3/include/cobalt/wrappers.h:21:0,
>>                   from
>> /home/sdf/xenobuild/imx-xenomai/xenomai-3/include/cobalt/time.h:26,
>>                   from
>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/sys/types.h:132,
>>                   from
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:19:
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>> â?~open_heapsâ?T:
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>> dereferencing pointer to incomplete type â?~struct shared_heapâ?T
>>     heap = container_of(obj, struct shared_heap, memspec);
>>            ^
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>> initialization from incompatible pointer type
>> [-Werror=incompatible-pointer-types]
>>     heap = container_of(obj, struct shared_heap, memspec);
>>            ^
>> In file included from
>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/string.h:32:0,
>>                   from
>> ../../../../../xenomai-3/include/copperplate/heapobj.h:24,
>>                   from
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:27:
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:35: error:
>> invalid use of undefined type â?~struct shared_heapâ?T
>>     heap = container_of(obj, struct shared_heap, memspec);
>>                                     ^
>> ------------------------------------------------------------------------
>> This is with "struct shared_heap" changed to "struct shared_heap_memory":
>>
>> make[3]: Entering directory
>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>    CC       regd.o
>>    CC       fs-common.o
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>> âopen_heapsâ:
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:249:17: error:
>> âstruct shared_heap_memoryâ has no member named âubytesâ
>>     p->used = heap->ubytes;
>>                   ^
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:250:18: error:
>> âstruct shared_heap_memoryâ has no member named âtotalâ
>>     p->total = heap->total;
>>                    ^
>> Makefile:472: recipe for target 'fs-common.o' failed
>> make[3]: *** [fs-common.o] Error 1
>>
> This may help:
>
> diff --git a/lib/copperplate/regd/fs-common.c b/lib/copperplate/regd/fs-common.c
> index ef2e6dae9..e0bdd3d04 100644
> --- a/lib/copperplate/regd/fs-common.c
> +++ b/lib/copperplate/regd/fs-common.c
> @@ -211,8 +211,8 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>   {
>   	struct sysgroup_memspec *obj, *tmp;
>   	struct heap_data *heap_data, *p;
> +	struct shared_heap_memory *heap;
>   	struct fsobstack *o = priv;
> -	struct shared_heap *heap;
>   	int ret, count, len = 0;
>   
>   	ret = heapobj_bind_session(__copperplate_setup_data.session_label);
> @@ -244,10 +244,10 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>   	for_each_sysgroup(obj, tmp, heap) {
>   		if (p - heap_data >= count)
>   			break;
> -		heap = container_of(obj, struct shared_heap, memspec);
> +		heap = container_of(obj, struct shared_heap_memory, memspec);
>   		namecpy(p->name, heap->name);
> -		p->used = heap->ubytes;
> -		p->total = heap->total;
> +		p->used = heap->used_size;
> +		p->total = heap->usable_size;
>   		p++;
>   	}
>
Philippe,

Thanks for that update, I see that it made it into the -next branch.  
The build went fine
and I updated my SDK as well with the new heapobj headers, etc.  But now 
it appears as though
g++ is unhappy with casting between void* and heapobj*, heap_memory*, 
etc that I'm guessing has
to do with alchemy headers ref'ing the heap pointers using void *. I'm 
ignoring this for now, focusing on the apps which are in C, which is 95% 
of what I have, and will look at using
-fpermissive or #pragma's to fix the C++ issues later.

This is my first attempt with pshared since the new allocator went in.  
I've always launched
sysregd during system startup, like this:

sysregd --daemon --root /var/run/xenomai/root/opus --linger

Here's a traceback which happens with our rt_queue reader app, it only 
happens when the
session already exists, not when it's the first process (except sysregd) 
in the session:

./mqueue-reader
2165 Segmentation fault      (core dumped) ./mqueue-reader "$@"

gdb session with core file:

Core was generated by `./mqueue-readerE --mem-pool-size=32M --session=opus'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x76ee8704 in ?? ()
[Current thread is 1 (LWP 2104)]
(gdb) bt
#0  0x76ee8704 in ?? ()
#1  0x76fc38c0 in shavl_search_inner (delta=0x7ec7b9f0, n=0x7ec7ba04, 
avl=0x7445c8f4) at ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
#2  shavl_search_nearest (dir=1, node=0x7ec7ba04, avl=0x7445c8f4) at 
../../../../xenomai-3/include/boilerplate/avl-inner.h:393
#3  shavl_search_ge (node=0x7ec7ba04, avl=0x7445c8f4) at 
../../../../xenomai-3/include/boilerplate/avl-inner.h:411
#4  find_suitable_range (size=1536, ext=0x7445c8b8) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:204
#5  reserve_page_range (size=1536, ext=0x7445c8b8) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:215
#6  add_free_range (heap=0x2, heap@entry=0x7445c000, bsize=2127018500, 
bsize@entry=1536, log2size=1950728192, log2size@entry=0)
     at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:422
#7  0x76fc3b9c in sheapmem_alloc (heap=0x7445c000, size=size@entry=1272) 
at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:543
#8  0x76fc49d0 in xnmalloc (size=size@entry=1272) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:1177
#9  0x76fc18d4 in __threadobj_alloc 
(tcb_struct_size=tcb_struct_size@entry=248, 
wait_union_size=wait_union_size@entry=1024,
     thobj_offset=thobj_offset@entry=0) at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1118
#10 0x76fc27a0 in main_overlay () at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1787
#11 threadobj_pkg_init (anon_session=anon_session@entry=0) at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1818
#12 0x76fbfa1c in copperplate_init () at 
../../../../xenomai-3/lib/copperplate/init.c:225
#13 0x76f9e534 in __xenomai_init (argcp=0x7ec7bbfc, 
argcp@entry=0x7ec7bc44, argvp=0x103d428, argvp@entry=0x7ec7bc40,
     me=me@entry=0x76fa6460 "program") at 
../../../../xenomai-3/lib/boilerplate/setup.c:601
#14 0x76f9ecf8 in xenomai_init (argcp=0x7ec7bc44, 
argcp@entry=0x7ec7bc3c, argvp=argvp@entry=0x7ec7bc40)
     at ../../../../xenomai-3/lib/boilerplate/setup.c:656
#15 0x00010f0c in call_init (argvp=0x7ec7bc40, argcp=0x7ec7bc3c) at 
../../../../../xenomai-3/lib/boilerplate/init/bootstrap.c:105
#16 xenomai_bootstrap () at 
../../../../../xenomai-3/lib/boilerplate/init/bootstrap.c:169
#17 0x00011864 in __libc_csu_init (argc=3, argv=0x7ec7bdd4, 
envp=0x7ec7bde4) at /usr/src/debug/glibc/2.23-r0/git/csu/elf-init.c:88
#18 0x41408088 in __libc_start_main () from /lib/libc.so.6
#19 0x00010f68 in _start () at ../sysdeps/arm/start.S:124
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

The mqueue-reader app is larger than it should be so if this traceback 
isn't enough to lead
you to a solution and you need some test code I will trim it down to 
something much smaller that
still shows the issue and send that along.

Thanks in advance,
Steve

Re: [Xenomai] Performance issue with memory allocators

[Philippe Gerum]

On 06/12/2018 09:15 PM, Steve Freyder wrote:
> On 6/12/2018 1:39 AM, Philippe Gerum wrote:
>> On 06/11/2018 10:37 PM, Steve Freyder wrote:
>>> On 6/8/2018 7:48 AM, Philippe Gerum wrote:
>>>> On 05/20/2018 08:15 PM, Philippe Gerum wrote:
>>>>> On 04/27/2018 08:23 PM, Philippe Gerum wrote:
>>>>>> After a comprehensive series of tests, it is now obvious that
>>>>>> heapobj-pshared has gruesome performances when releasing bursts of
>>>>>> small-sized objects. Allocation of large objects is not particularly
>>>>>> efficient either, although not as bad. The Cobalt core allocator is
>>>>>> affected by the same bug too.
>>>>>>
>>>>>> Those new tests are available from the wip/heapmem branch, they are
>>>>>> based on the smokey framework (e.g. smokey --run=memory_*
>>>>>> --verbose=2).
>>>>>>
>>>>>> Work is ongoing into the wip/heapmem branch to fix the mess, first
>>>>>> addressing the issue with the core (in-kernel) allocator,
>>>>>> heapobj-pshared next. Please note that wip/heapmem is a development
>>>>>> branch I'm working on for testing improvements - hopefully - to the
>>>>>> memory allocators, which should not be used in production.
>>>>>>
>>>>>> I would strongly recommend that users running single-process
>>>>>> applications - i.e. not needing the shared heap feature - make
>>>>>> sure to
>>>>>> turn off heapobj-pshared when configuring, by passing
>>>>>> --disable-pshared
>>>>>> to the configuration script (or leave the feature to the default
>>>>>> off state).
>>>>>>
>>>>>> As the benchmark figures show when running 'memory_pshared', the
>>>>>> shared
>>>>> memory allocator still needs to be fixed. This is next on my list.
>>>> The pshared allocator is now fixed in the next branch.
>>>>
>>> I tried to build with the -next branch, I'm not sure if I'm doing
>>> something wrong, but I'm getting the errors below.  I thought perhaps
>>> the "struct shared_heap" should have been "struct shared_heap_memory",
>>> so I made that change, and that resulted in the second set of errors
>>> shown below that, then I decided time to post to the group:
>>>
>>> Thanks,
>>> Steve
>>>
>>> ------------------------------------------------------------------------
>>>
>>> make[3]: Entering directory
>>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>>    CC       regd.o
>>>    CC       fs-common.o
>>> In file included from
>>> ../../../../../xenomai-3/include/cobalt/wrappers.h:21:0,
>>>                   from
>>> /home/sdf/xenobuild/imx-xenomai/xenomai-3/include/cobalt/time.h:26,
>>>                   from
>>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/sys/types.h:132,
>>>
>>>                   from
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:19:
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>>> â?~open_heapsâ?T:
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>>> dereferencing pointer to incomplete type â?~struct shared_heapâ?T
>>>     heap = container_of(obj, struct shared_heap, memspec);
>>>            ^
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>>> initialization from incompatible pointer type
>>> [-Werror=incompatible-pointer-types]
>>>     heap = container_of(obj, struct shared_heap, memspec);
>>>            ^
>>> In file included from
>>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/string.h:32:0,
>>>
>>>                   from
>>> ../../../../../xenomai-3/include/copperplate/heapobj.h:24,
>>>                   from
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:27:
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:35: error:
>>> invalid use of undefined type â?~struct shared_heapâ?T
>>>     heap = container_of(obj, struct shared_heap, memspec);
>>>                                     ^
>>> ------------------------------------------------------------------------
>>> This is with "struct shared_heap" changed to "struct
>>> shared_heap_memory":
>>>
>>> make[3]: Entering directory
>>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>>    CC       regd.o
>>>    CC       fs-common.o
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>>> âopen_heapsâ:
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:249:17: error:
>>> âstruct shared_heap_memoryâ has no member named âubytesâ
>>>     p->used = heap->ubytes;
>>>                   ^
>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:250:18: error:
>>> âstruct shared_heap_memoryâ has no member named âtotalâ
>>>     p->total = heap->total;
>>>                    ^
>>> Makefile:472: recipe for target 'fs-common.o' failed
>>> make[3]: *** [fs-common.o] Error 1
>>>
>> This may help:
>>
>> diff --git a/lib/copperplate/regd/fs-common.c
>> b/lib/copperplate/regd/fs-common.c
>> index ef2e6dae9..e0bdd3d04 100644
>> --- a/lib/copperplate/regd/fs-common.c
>> +++ b/lib/copperplate/regd/fs-common.c
>> @@ -211,8 +211,8 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>>   {
>>       struct sysgroup_memspec *obj, *tmp;
>>       struct heap_data *heap_data, *p;
>> +    struct shared_heap_memory *heap;
>>       struct fsobstack *o = priv;
>> -    struct shared_heap *heap;
>>       int ret, count, len = 0;
>>         ret =
>> heapobj_bind_session(__copperplate_setup_data.session_label);
>> @@ -244,10 +244,10 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>>       for_each_sysgroup(obj, tmp, heap) {
>>           if (p - heap_data >= count)
>>               break;
>> -        heap = container_of(obj, struct shared_heap, memspec);
>> +        heap = container_of(obj, struct shared_heap_memory, memspec);
>>           namecpy(p->name, heap->name);
>> -        p->used = heap->ubytes;
>> -        p->total = heap->total;
>> +        p->used = heap->used_size;
>> +        p->total = heap->usable_size;
>>           p++;
>>       }
>>
> Philippe,
> 
> Thanks for that update, I see that it made it into the -next branch. 
> The build went fine
> and I updated my SDK as well with the new heapobj headers, etc.  But now
> it appears as though
> g++ is unhappy with casting between void* and heapobj*, heap_memory*,
> etc that I'm guessing has
> to do with alchemy headers ref'ing the heap pointers using void *. I'm
> ignoring this for now, focusing on the apps which are in C, which is 95%
> of what I have, and will look at using
> -fpermissive or #pragma's to fix the C++ issues later.
> 

I fixed this in the repo, only tested with a trivial C++ app pulling in
the headers though. Please let me know if that works for you too.

> This is my first attempt with pshared since the new allocator went in. 
> I've always launched
> sysregd during system startup, like this:
> 
> sysregd --daemon --root /var/run/xenomai/root/opus --linger
> 
> Here's a traceback which happens with our rt_queue reader app, it only
> happens when the
> session already exists, not when it's the first process (except sysregd)
> in the session:
> 
> ./mqueue-reader
> 2165 Segmentation fault      (core dumped) ./mqueue-reader "$@"
> 
> gdb session with core file:
> 
> Core was generated by `./mqueue-readerE --mem-pool-size=32M
> --session=opus'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x76ee8704 in ?? ()
> [Current thread is 1 (LWP 2104)]
> (gdb) bt
> #0  0x76ee8704 in ?? ()
> #1  0x76fc38c0 in shavl_search_inner (delta=0x7ec7b9f0, n=0x7ec7ba04,
> avl=0x7445c8f4) at
> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
> #2  shavl_search_nearest (dir=1, node=0x7ec7ba04, avl=0x7445c8f4) at
> ../../../../xenomai-3/include/boilerplate/avl-inner.h:393
> #3  shavl_search_ge (node=0x7ec7ba04, avl=0x7445c8f4) at
> ../../../../xenomai-3/include/boilerplate/avl-inner.h:411
> #4  find_suitable_range (size=1536, ext=0x7445c8b8) at
> ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:204
> #5  reserve_page_range (size=1536, ext=0x7445c8b8) at
> ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:215
> #6  add_free_range (heap=0x2, heap@entry=0x7445c000, bsize=2127018500,
> bsize@entry=1536, log2size=1950728192, log2size@entry=0)
>     at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:422
> #7  0x76fc3b9c in sheapmem_alloc (heap=0x7445c000, size=size@entry=1272)
> at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:543
>

Going from heap=0x7445c000 with size=1272 to heap=0x2 with
size=2127018500 looks weird, and won't work obviously. Any chance a
stack overflow would be at work here?

> The mqueue-reader app is larger than it should be so if this traceback
> isn't enough to lead
> you to a solution and you need some test code I will trim it down to
> something much smaller that
> still shows the issue and send that along.
> 

You may want to run the new stress test exercising all of the memory
allocators, specifically for the heapmem-pshared:

$ smokey --run=memory_pshared --verbose=2

If that one runs properly on your setup, I would suspect an issue local
to the app (e.g. such as a stack overflow).

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder]

On 6/13/2018 2:06 AM, Philippe Gerum wrote:
> On 06/12/2018 09:15 PM, Steve Freyder wrote:
>> On 6/12/2018 1:39 AM, Philippe Gerum wrote:
>>> On 06/11/2018 10:37 PM, Steve Freyder wrote:
>>>> On 6/8/2018 7:48 AM, Philippe Gerum wrote:
>>>>> On 05/20/2018 08:15 PM, Philippe Gerum wrote:
>>>>>> On 04/27/2018 08:23 PM, Philippe Gerum wrote:
>>>>>>> After a comprehensive series of tests, it is now obvious that
>>>>>>> heapobj-pshared has gruesome performances when releasing bursts of
>>>>>>> small-sized objects. Allocation of large objects is not particularly
>>>>>>> efficient either, although not as bad. The Cobalt core allocator is
>>>>>>> affected by the same bug too.
>>>>>>>
>>>>>>> Those new tests are available from the wip/heapmem branch, they are
>>>>>>> based on the smokey framework (e.g. smokey --run=memory_*
>>>>>>> --verbose=2).
>>>>>>>
>>>>>>> Work is ongoing into the wip/heapmem branch to fix the mess, first
>>>>>>> addressing the issue with the core (in-kernel) allocator,
>>>>>>> heapobj-pshared next. Please note that wip/heapmem is a development
>>>>>>> branch I'm working on for testing improvements - hopefully - to the
>>>>>>> memory allocators, which should not be used in production.
>>>>>>>
>>>>>>> I would strongly recommend that users running single-process
>>>>>>> applications - i.e. not needing the shared heap feature - make
>>>>>>> sure to
>>>>>>> turn off heapobj-pshared when configuring, by passing
>>>>>>> --disable-pshared
>>>>>>> to the configuration script (or leave the feature to the default
>>>>>>> off state).
>>>>>>>
>>>>>>> As the benchmark figures show when running 'memory_pshared', the
>>>>>>> shared
>>>>>> memory allocator still needs to be fixed. This is next on my list.
>>>>> The pshared allocator is now fixed in the next branch.
>>>>>
>>>> I tried to build with the -next branch, I'm not sure if I'm doing
>>>> something wrong, but I'm getting the errors below.  I thought perhaps
>>>> the "struct shared_heap" should have been "struct shared_heap_memory",
>>>> so I made that change, and that resulted in the second set of errors
>>>> shown below that, then I decided time to post to the group:
>>>>
>>>> Thanks,
>>>> Steve
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> make[3]: Entering directory
>>>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>>>     CC       regd.o
>>>>     CC       fs-common.o
>>>> In file included from
>>>> ../../../../../xenomai-3/include/cobalt/wrappers.h:21:0,
>>>>                    from
>>>> /home/sdf/xenobuild/imx-xenomai/xenomai-3/include/cobalt/time.h:26,
>>>>                    from
>>>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/sys/types.h:132,
>>>>
>>>>                    from
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:19:
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>>>> â?~open_heapsâ?T:
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>>>> dereferencing pointer to incomplete type â?~struct shared_heapâ?T
>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>             ^
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>>>> initialization from incompatible pointer type
>>>> [-Werror=incompatible-pointer-types]
>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>             ^
>>>> In file included from
>>>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/string.h:32:0,
>>>>
>>>>                    from
>>>> ../../../../../xenomai-3/include/copperplate/heapobj.h:24,
>>>>                    from
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:27:
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:35: error:
>>>> invalid use of undefined type â?~struct shared_heapâ?T
>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>                                      ^
>>>> ------------------------------------------------------------------------
>>>> This is with "struct shared_heap" changed to "struct
>>>> shared_heap_memory":
>>>>
>>>> make[3]: Entering directory
>>>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>>>     CC       regd.o
>>>>     CC       fs-common.o
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>>>> âopen_heapsâ:
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:249:17: error:
>>>> âstruct shared_heap_memoryâ has no member named âubytesâ
>>>>      p->used = heap->ubytes;
>>>>                    ^
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:250:18: error:
>>>> âstruct shared_heap_memoryâ has no member named âtotalâ
>>>>      p->total = heap->total;
>>>>                     ^
>>>> Makefile:472: recipe for target 'fs-common.o' failed
>>>> make[3]: *** [fs-common.o] Error 1
>>>>
>>> This may help:
>>>
>>> diff --git a/lib/copperplate/regd/fs-common.c
>>> b/lib/copperplate/regd/fs-common.c
>>> index ef2e6dae9..e0bdd3d04 100644
>>> --- a/lib/copperplate/regd/fs-common.c
>>> +++ b/lib/copperplate/regd/fs-common.c
>>> @@ -211,8 +211,8 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>>>    {
>>>        struct sysgroup_memspec *obj, *tmp;
>>>        struct heap_data *heap_data, *p;
>>> +    struct shared_heap_memory *heap;
>>>        struct fsobstack *o = priv;
>>> -    struct shared_heap *heap;
>>>        int ret, count, len = 0;
>>>          ret =
>>> heapobj_bind_session(__copperplate_setup_data.session_label);
>>> @@ -244,10 +244,10 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>>>        for_each_sysgroup(obj, tmp, heap) {
>>>            if (p - heap_data >= count)
>>>                break;
>>> -        heap = container_of(obj, struct shared_heap, memspec);
>>> +        heap = container_of(obj, struct shared_heap_memory, memspec);
>>>            namecpy(p->name, heap->name);
>>> -        p->used = heap->ubytes;
>>> -        p->total = heap->total;
>>> +        p->used = heap->used_size;
>>> +        p->total = heap->usable_size;
>>>            p++;
>>>        }
>>>
>> Philippe,
>>
>> Thanks for that update, I see that it made it into the -next branch.
>> The build went fine
>> and I updated my SDK as well with the new heapobj headers, etc.  But now
>> it appears as though
>> g++ is unhappy with casting between void* and heapobj*, heap_memory*,
>> etc that I'm guessing has
>> to do with alchemy headers ref'ing the heap pointers using void *. I'm
>> ignoring this for now, focusing on the apps which are in C, which is 95%
>> of what I have, and will look at using
>> -fpermissive or #pragma's to fix the C++ issues later.
>>
> I fixed this in the repo, only tested with a trivial C++ app pulling in
> the headers though. Please let me know if that works for you too.
>
>> This is my first attempt with pshared since the new allocator went in.
>> I've always launched
>> sysregd during system startup, like this:
>>
>> sysregd --daemon --root /var/run/xenomai/root/opus --linger
>>
>> Here's a traceback which happens with our rt_queue reader app, it only
>> happens when the
>> session already exists, not when it's the first process (except sysregd)
>> in the session:
>>
>> ./mqueue-reader
>> 2165 Segmentation fault      (core dumped) ./mqueue-reader "$@"
>>
>> gdb session with core file:
>>
>> Core was generated by `./mqueue-readerE --mem-pool-size=32M
>> --session=opus'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  0x76ee8704 in ?? ()
>> [Current thread is 1 (LWP 2104)]
>> (gdb) bt
>> #0  0x76ee8704 in ?? ()
>> #1  0x76fc38c0 in shavl_search_inner (delta=0x7ec7b9f0, n=0x7ec7ba04,
>> avl=0x7445c8f4) at
>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
>> #2  shavl_search_nearest (dir=1, node=0x7ec7ba04, avl=0x7445c8f4) at
>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:393
>> #3  shavl_search_ge (node=0x7ec7ba04, avl=0x7445c8f4) at
>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:411
>> #4  find_suitable_range (size=1536, ext=0x7445c8b8) at
>> ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:204
>> #5  reserve_page_range (size=1536, ext=0x7445c8b8) at
>> ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:215
>> #6  add_free_range (heap=0x2, heap@entry=0x7445c000, bsize=2127018500,
>> bsize@entry=1536, log2size=1950728192, log2size@entry=0)
>>      at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:422
>> #7  0x76fc3b9c in sheapmem_alloc (heap=0x7445c000, size=size@entry=1272)
>> at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:543
>>
> Going from heap=0x7445c000 with size=1272 to heap=0x2 with
> size=2127018500 looks weird, and won't work obviously. Any chance a
> stack overflow would be at work here?
>
>> The mqueue-reader app is larger than it should be so if this traceback
>> isn't enough to lead
>> you to a solution and you need some test code I will trim it down to
>> something much smaller that
>> still shows the issue and send that along.
>>
> You may want to run the new stress test exercising all of the memory
> allocators, specifically for the heapmem-pshared:
>
> $ smokey --run=memory_pshared --verbose=2
>
> If that one runs properly on your setup, I would suspect an issue local
> to the app (e.g. such as a stack overflow).
>
Philippe,

First, thanks for the update on the g++ header issue, my compiles are 
all clean now.

I rebuilt my program with a 512K stack, same problem with the SEGV.

Tried something different, without running my application at all, I ran 
this script:

ulimit -c unlimited
smokey --run=memory_pshared --verbose=2 --mem-pool-size=32M --session=foo

it ran fine, but if I ran it a second time, without waiting for the 
first invocation to
finish, then I got the same SEGV that I previously was seeing with my 
program and the
traceback looks the same.

It's as if there's an initialization issue when you're not the creator 
of the
pshared heap.

Re: [Xenomai] Performance issue with memory allocators

[Philippe Gerum]

On 06/14/2018 12:53 AM, Steve Freyder wrote:
> On 6/13/2018 2:06 AM, Philippe Gerum wrote:
>> On 06/12/2018 09:15 PM, Steve Freyder wrote:
>>> On 6/12/2018 1:39 AM, Philippe Gerum wrote:
>>>> On 06/11/2018 10:37 PM, Steve Freyder wrote:
>>>>> On 6/8/2018 7:48 AM, Philippe Gerum wrote:
>>>>>> On 05/20/2018 08:15 PM, Philippe Gerum wrote:
>>>>>>> On 04/27/2018 08:23 PM, Philippe Gerum wrote:
>>>>>>>> After a comprehensive series of tests, it is now obvious that
>>>>>>>> heapobj-pshared has gruesome performances when releasing bursts of
>>>>>>>> small-sized objects. Allocation of large objects is not
>>>>>>>> particularly
>>>>>>>> efficient either, although not as bad. The Cobalt core allocator is
>>>>>>>> affected by the same bug too.
>>>>>>>>
>>>>>>>> Those new tests are available from the wip/heapmem branch, they are
>>>>>>>> based on the smokey framework (e.g. smokey --run=memory_*
>>>>>>>> --verbose=2).
>>>>>>>>
>>>>>>>> Work is ongoing into the wip/heapmem branch to fix the mess, first
>>>>>>>> addressing the issue with the core (in-kernel) allocator,
>>>>>>>> heapobj-pshared next. Please note that wip/heapmem is a development
>>>>>>>> branch I'm working on for testing improvements - hopefully - to the
>>>>>>>> memory allocators, which should not be used in production.
>>>>>>>>
>>>>>>>> I would strongly recommend that users running single-process
>>>>>>>> applications - i.e. not needing the shared heap feature - make
>>>>>>>> sure to
>>>>>>>> turn off heapobj-pshared when configuring, by passing
>>>>>>>> --disable-pshared
>>>>>>>> to the configuration script (or leave the feature to the default
>>>>>>>> off state).
>>>>>>>>
>>>>>>>> As the benchmark figures show when running 'memory_pshared', the
>>>>>>>> shared
>>>>>>> memory allocator still needs to be fixed. This is next on my list.
>>>>>> The pshared allocator is now fixed in the next branch.
>>>>>>
>>>>> I tried to build with the -next branch, I'm not sure if I'm doing
>>>>> something wrong, but I'm getting the errors below.  I thought perhaps
>>>>> the "struct shared_heap" should have been "struct shared_heap_memory",
>>>>> so I made that change, and that resulted in the second set of errors
>>>>> shown below that, then I decided time to post to the group:
>>>>>
>>>>> Thanks,
>>>>> Steve
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> make[3]: Entering directory
>>>>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>>>>     CC       regd.o
>>>>>     CC       fs-common.o
>>>>> In file included from
>>>>> ../../../../../xenomai-3/include/cobalt/wrappers.h:21:0,
>>>>>                    from
>>>>> /home/sdf/xenobuild/imx-xenomai/xenomai-3/include/cobalt/time.h:26,
>>>>>                    from
>>>>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/sys/types.h:132,
>>>>>
>>>>>
>>>>>                    from
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:19:
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>>>>> â?~open_heapsâ?T:
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10:
>>>>> error:
>>>>> dereferencing pointer to incomplete type â?~struct shared_heapâ?T
>>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>>             ^
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10:
>>>>> error:
>>>>> initialization from incompatible pointer type
>>>>> [-Werror=incompatible-pointer-types]
>>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>>             ^
>>>>> In file included from
>>>>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/string.h:32:0,
>>>>>
>>>>>
>>>>>                    from
>>>>> ../../../../../xenomai-3/include/copperplate/heapobj.h:24,
>>>>>                    from
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:27:
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:35:
>>>>> error:
>>>>> invalid use of undefined type â?~struct shared_heapâ?T
>>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>>                                      ^
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> This is with "struct shared_heap" changed to "struct
>>>>> shared_heap_memory":
>>>>>
>>>>> make[3]: Entering directory
>>>>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>>>>     CC       regd.o
>>>>>     CC       fs-common.o
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>>>>> âopen_heapsâ:
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:249:17:
>>>>> error:
>>>>> âstruct shared_heap_memoryâ has no member named âubytesâ
>>>>>      p->used = heap->ubytes;
>>>>>                    ^
>>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:250:18:
>>>>> error:
>>>>> âstruct shared_heap_memoryâ has no member named âtotalâ
>>>>>      p->total = heap->total;
>>>>>                     ^
>>>>> Makefile:472: recipe for target 'fs-common.o' failed
>>>>> make[3]: *** [fs-common.o] Error 1
>>>>>
>>>> This may help:
>>>>
>>>> diff --git a/lib/copperplate/regd/fs-common.c
>>>> b/lib/copperplate/regd/fs-common.c
>>>> index ef2e6dae9..e0bdd3d04 100644
>>>> --- a/lib/copperplate/regd/fs-common.c
>>>> +++ b/lib/copperplate/regd/fs-common.c
>>>> @@ -211,8 +211,8 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>>>>    {
>>>>        struct sysgroup_memspec *obj, *tmp;
>>>>        struct heap_data *heap_data, *p;
>>>> +    struct shared_heap_memory *heap;
>>>>        struct fsobstack *o = priv;
>>>> -    struct shared_heap *heap;
>>>>        int ret, count, len = 0;
>>>>          ret =
>>>> heapobj_bind_session(__copperplate_setup_data.session_label);
>>>> @@ -244,10 +244,10 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>>>>        for_each_sysgroup(obj, tmp, heap) {
>>>>            if (p - heap_data >= count)
>>>>                break;
>>>> -        heap = container_of(obj, struct shared_heap, memspec);
>>>> +        heap = container_of(obj, struct shared_heap_memory, memspec);
>>>>            namecpy(p->name, heap->name);
>>>> -        p->used = heap->ubytes;
>>>> -        p->total = heap->total;
>>>> +        p->used = heap->used_size;
>>>> +        p->total = heap->usable_size;
>>>>            p++;
>>>>        }
>>>>
>>> Philippe,
>>>
>>> Thanks for that update, I see that it made it into the -next branch.
>>> The build went fine
>>> and I updated my SDK as well with the new heapobj headers, etc.  But now
>>> it appears as though
>>> g++ is unhappy with casting between void* and heapobj*, heap_memory*,
>>> etc that I'm guessing has
>>> to do with alchemy headers ref'ing the heap pointers using void *. I'm
>>> ignoring this for now, focusing on the apps which are in C, which is 95%
>>> of what I have, and will look at using
>>> -fpermissive or #pragma's to fix the C++ issues later.
>>>
>> I fixed this in the repo, only tested with a trivial C++ app pulling in
>> the headers though. Please let me know if that works for you too.
>>
>>> This is my first attempt with pshared since the new allocator went in.
>>> I've always launched
>>> sysregd during system startup, like this:
>>>
>>> sysregd --daemon --root /var/run/xenomai/root/opus --linger
>>>
>>> Here's a traceback which happens with our rt_queue reader app, it only
>>> happens when the
>>> session already exists, not when it's the first process (except sysregd)
>>> in the session:
>>>
>>> ./mqueue-reader
>>> 2165 Segmentation fault      (core dumped) ./mqueue-reader "$@"
>>>
>>> gdb session with core file:
>>>
>>> Core was generated by `./mqueue-readerE --mem-pool-size=32M
>>> --session=opus'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0  0x76ee8704 in ?? ()
>>> [Current thread is 1 (LWP 2104)]
>>> (gdb) bt
>>> #0  0x76ee8704 in ?? ()
>>> #1  0x76fc38c0 in shavl_search_inner (delta=0x7ec7b9f0, n=0x7ec7ba04,
>>> avl=0x7445c8f4) at
>>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
>>> #2  shavl_search_nearest (dir=1, node=0x7ec7ba04, avl=0x7445c8f4) at
>>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:393
>>> #3  shavl_search_ge (node=0x7ec7ba04, avl=0x7445c8f4) at
>>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:411
>>> #4  find_suitable_range (size=1536, ext=0x7445c8b8) at
>>> ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:204
>>> #5  reserve_page_range (size=1536, ext=0x7445c8b8) at
>>> ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:215
>>> #6  add_free_range (heap=0x2, heap@entry=0x7445c000, bsize=2127018500,
>>> bsize@entry=1536, log2size=1950728192, log2size@entry=0)
>>>      at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:422
>>> #7  0x76fc3b9c in sheapmem_alloc (heap=0x7445c000, size=size@entry=1272)
>>> at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:543
>>>
>> Going from heap=0x7445c000 with size=1272 to heap=0x2 with
>> size=2127018500 looks weird, and won't work obviously. Any chance a
>> stack overflow would be at work here?
>>
>>> The mqueue-reader app is larger than it should be so if this traceback
>>> isn't enough to lead
>>> you to a solution and you need some test code I will trim it down to
>>> something much smaller that
>>> still shows the issue and send that along.
>>>
>> You may want to run the new stress test exercising all of the memory
>> allocators, specifically for the heapmem-pshared:
>>
>> $ smokey --run=memory_pshared --verbose=2
>>
>> If that one runs properly on your setup, I would suspect an issue local
>> to the app (e.g. such as a stack overflow).
>>
> Philippe,
> 
> First, thanks for the update on the g++ header issue, my compiles are
> all clean now.
> 
> I rebuilt my program with a 512K stack, same problem with the SEGV.
> 
> Tried something different, without running my application at all, I ran
> this script:
> 
> ulimit -c unlimited
> smokey --run=memory_pshared --verbose=2 --mem-pool-size=32M --session=foo
> 
> it ran fine, but if I ran it a second time, without waiting for the
> first invocation to
> finish, then I got the same SEGV that I previously was seeing with my
> program and the
> traceback looks the same.
> 
> It's as if there's an initialization issue when you're not the creator
> of the
> pshared heap.
> 
> 

Yeah, this can't work just yet. Implementation blunder. I'll follow up
on this.

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Philippe Gerum]

On 06/14/2018 03:10 PM, Philippe Gerum wrote:
> On 06/14/2018 12:53 AM, Steve Freyder wrote:
>> I rebuilt my program with a 512K stack, same problem with the SEGV.
>>
>> Tried something different, without running my application at all, I ran
>> this script:
>>
>> ulimit -c unlimited
>> smokey --run=memory_pshared --verbose=2 --mem-pool-size=32M --session=foo
>>
>> it ran fine, but if I ran it a second time, without waiting for the
>> first invocation to
>> finish, then I got the same SEGV that I previously was seeing with my
>> program and the
>> traceback looks the same.
>>
>> It's as if there's an initialization issue when you're not the creator
>> of the
>> pshared heap.
>>
>>
> 
> Yeah, this can't work just yet. Implementation blunder. I'll follow up
> on this.
> 

Could you check if the situation gets any better with this change in?

http://git.xenomai.org/xenomai-3.git/commit/?h=next&id=7e3d69ae6c9235289a8984b609ae2400ab950f17

Thanks,

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder]

On 6/14/2018 10:21 AM, Philippe Gerum wrote:
> On 06/14/2018 03:10 PM, Philippe Gerum wrote:
>> On 06/14/2018 12:53 AM, Steve Freyder wrote:
>>> I rebuilt my program with a 512K stack, same problem with the SEGV.
>>>
>>> Tried something different, without running my application at all, I ran
>>> this script:
>>>
>>> ulimit -c unlimited
>>> smokey --run=memory_pshared --verbose=2 --mem-pool-size=32M --session=foo
>>>
>>> it ran fine, but if I ran it a second time, without waiting for the
>>> first invocation to
>>> finish, then I got the same SEGV that I previously was seeing with my
>>> program and the
>>> traceback looks the same.
>>>
>>> It's as if there's an initialization issue when you're not the creator
>>> of the
>>> pshared heap.
>>>
>>>
>> Yeah, this can't work just yet. Implementation blunder. I'll follow up
>> on this.
>>
> Could you check if the situation gets any better with this change in?
>
> http://git.xenomai.org/xenomai-3.git/commit/?h=next&id=7e3d69ae6c9235289a8984b609ae2400ab950f17
>
> Thanks,
>
Philippe,

This looks pretty similar.  At first I thought my patching didn't fly, 
but I double-checked
my work and although the size of the "smokey" binary didn't change, the 
size of
libcopperplate DID change so the patch is there.

Core was generated by `smokey --run=memory_pshared --verbose=2 
--mem-pool-size=32M --session=foo'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  compare_range_by_size (r=0xfffcb000, l=0x741f38ec) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:638
638     ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c: No such 
file or directory.
[Current thread is 1 (LWP 800)]
(gdb) bt
#0  compare_range_by_size (r=0xfffcb000, l=0x741f38ec) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:638
#1  search_range_by_size (avl=avl@entry=0x741f38ec, 
node=node@entry=0x7eca29d4, pdelta=0x7eca29c0,
     pdelta@entry=0x76f418fc <add_free_range+216>, dir=dir@entry=0) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:640
#2  0x76f418fc in shavl_search_inner (ops=0x76f56630 <size_search_ops>, 
delta=0x76f418fc <add_free_range+216>, n=0x7eca29d4,
     avl=0x741f38ec) at 
../../../../xenomai-3/include/boilerplate/avl-inner.h:285
#3  shavl_search_nearest (ops=0x76f56630 <size_search_ops>, dir=1, 
node=0x7eca29d4, avl=0x741f38ec)
     at ../../../../xenomai-3/include/boilerplate/avl-inner.h:395
#4  shavl_search_ge (ops=0x76f56630 <size_search_ops>, node=0x7eca29d4, 
avl=0x741f38ec)
     at ../../../../xenomai-3/include/boilerplate/avl-inner.h:415
#5  find_suitable_range (size=1536, ext=0x741f38b8) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:207
#6  reserve_page_range (size=1536, ext=0x741f38b8) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:219
#7  add_free_range (heap=0x7eca29c0, heap@entry=0x741f3000, 
bsize=1995793968, bsize@entry=1536, log2size=1948200960, log2size@entry=0)
     at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:432
#8  0x76f41be4 in sheapmem_alloc (heap=0x741f3000, size=size@entry=1272) 
at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:553
#9  0x76f42a44 in xnmalloc (size=size@entry=1272) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:1197
#10 0x76f3f8f4 in __threadobj_alloc 
(tcb_struct_size=tcb_struct_size@entry=248, 
wait_union_size=wait_union_size@entry=1024,
     thobj_offset=thobj_offset@entry=0) at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1118
#11 0x76f407c0 in main_overlay () at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1787
#12 threadobj_pkg_init (anon_session=anon_session@entry=0) at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1818
#13 0x76f3da3c in copperplate_init () at 
../../../../xenomai-3/lib/copperplate/init.c:225
#14 0x76f1c5ac in __xenomai_init (argcp=0x7eca2bcc, 
argcp@entry=0x7eca2c14, argvp=0x1a7430, argvp@entry=0x7eca2c10,
     me=me@entry=0x76f244e0 "program") at 
../../../../xenomai-3/lib/boilerplate/setup.c:601
#15 0x76f1cd70 in xenomai_init (argcp=0x7eca2c14, 
argcp@entry=0x7eca2c0c, argvp=argvp@entry=0x7eca2c10)
     at ../../../../xenomai-3/lib/boilerplate/setup.c:656
#16 0x000149f0 in call_init (argvp=0x7eca2c10, argcp=0x7eca2c0c) at 
../../../../../xenomai-3/lib/boilerplate/init/bootstrap.c:105
#17 xenomai_bootstrap () at 
../../../../../xenomai-3/lib/boilerplate/init/bootstrap.c:169
#18 0x0002a334 in __libc_csu_init (argc=5, argv=0x7eca2da4, 
envp=0x7eca2dbc) at /usr/src/debug/glibc/2.23-r0/git/csu/elf-init.c:88
#19 0x41408088 in __libc_start_main () from /lib/libc.so.6
#20 0x00014c94 in _start () at ../sysdeps/arm/start.S:124
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

Re: [Xenomai] Performance issue with memory allocators

[Philippe Gerum]

On 06/14/2018 06:41 PM, Steve Freyder wrote:
> #2  0x76f418fc in shavl_search_inner (ops=0x76f56630 <size_search_ops>,
> delta=0x76f418fc <add_free_range+216>, n=0x7eca29d4,
>     avl=0x741f38ec) at
> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
> #3  shavl_search_nearest (ops=0x76f56630 <size_search_ops>, dir=1,
> node=0x7eca29d4, avl=0x741f38ec)
>     at ../../../../xenomai-3/include/boilerplate/avl-inner.h:395
> #4  shavl_search_ge (ops=0x76f56630 <size_search_ops>, node=0x7eca29d4,
> avl=0x741f38ec)

The search operations parameter is last in the original code, but first
in your backtrace. Does this mean that you had conflicts there and fixed
them up manually? Since this is a new code, I'm unsure why you had such
conflicts in the first place.

Could you check with a pristine -next tree so that we may compare our
results from the same code base?

Thanks,

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder]

On 6/14/2018 11:52 AM, Philippe Gerum wrote:
> On 06/14/2018 06:41 PM, Steve Freyder wrote:
>> #2  0x76f418fc in shavl_search_inner (ops=0x76f56630 <size_search_ops>,
>> delta=0x76f418fc <add_free_range+216>, n=0x7eca29d4,
>>      avl=0x741f38ec) at
>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
>> #3  shavl_search_nearest (ops=0x76f56630 <size_search_ops>, dir=1,
>> node=0x7eca29d4, avl=0x741f38ec)
>>      at ../../../../xenomai-3/include/boilerplate/avl-inner.h:395
>> #4  shavl_search_ge (ops=0x76f56630 <size_search_ops>, node=0x7eca29d4,
>> avl=0x741f38ec)
> The search operations parameter is last in the original code, but first
> in your backtrace. Does this mean that you had conflicts there and fixed
> them up manually? Since this is a new code, I'm unsure why you had such
> conflicts in the first place.
>
> Could you check with a pristine -next tree so that we may compare our
> results from the same code base?
>
> Thanks,
>
Philippe,

I went back to do a full git clone, checkout, patch, rebuild, and I
found that when I fetched the -next branch that the patch had been
comitted already.  So I proceeded to skip to the build step, and then
retest the simultaneous execution of two instances of the smokey
memory_pshared test, but still got the same failure.

Are you seeing the test failure with your build?

I've never had any reason to doubt the validity of my SDK or build
procedures, but if your build is passing the test, something must be
wrong on my end.

Steve

Re: [Xenomai] Performance issue with memory allocators

[Philippe Gerum]

On 06/15/2018 07:23 PM, Steve Freyder wrote:
> On 6/14/2018 11:52 AM, Philippe Gerum wrote:
>> On 06/14/2018 06:41 PM, Steve Freyder wrote:
>>> #2  0x76f418fc in shavl_search_inner (ops=0x76f56630 <size_search_ops>,
>>> delta=0x76f418fc <add_free_range+216>, n=0x7eca29d4,
>>>      avl=0x741f38ec) at
>>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
>>> #3  shavl_search_nearest (ops=0x76f56630 <size_search_ops>, dir=1,
>>> node=0x7eca29d4, avl=0x741f38ec)
>>>      at ../../../../xenomai-3/include/boilerplate/avl-inner.h:395
>>> #4  shavl_search_ge (ops=0x76f56630 <size_search_ops>, node=0x7eca29d4,
>>> avl=0x741f38ec)
>> The search operations parameter is last in the original code, but first
>> in your backtrace. Does this mean that you had conflicts there and fixed
>> them up manually? Since this is a new code, I'm unsure why you had such
>> conflicts in the first place.
>>
>> Could you check with a pristine -next tree so that we may compare our
>> results from the same code base?
>>
>> Thanks,
>>
> Philippe,
> 
> I went back to do a full git clone, checkout, patch, rebuild, and I
> found that when I fetched the -next branch that the patch had been
> comitted already.  So I proceeded to skip to the build step, and then
> retest the simultaneous execution of two instances of the smokey
> memory_pshared test, but still got the same failure.
> 
> Are you seeing the test failure with your build?

No, that is why I've asked for a confirmation actually.

> 
> I've never had any reason to doubt the validity of my SDK or build
> procedures, but if your build is passing the test, something must be
> wrong on my end.
> 

Not necessarily, maybe I got lucky because of some setting. I'll follow
up on this when I have tested more configurations.

-- 
Philippe.

Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder]

On 6/15/2018 1:26 PM, Philippe Gerum wrote:
> On 06/15/2018 07:23 PM, Steve Freyder wrote:
>> On 6/14/2018 11:52 AM, Philippe Gerum wrote:
>>> On 06/14/2018 06:41 PM, Steve Freyder wrote:
>>>> #2  0x76f418fc in shavl_search_inner (ops=0x76f56630 <size_search_ops>,
>>>> delta=0x76f418fc <add_free_range+216>, n=0x7eca29d4,
>>>>       avl=0x741f38ec) at
>>>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
>>>> #3  shavl_search_nearest (ops=0x76f56630 <size_search_ops>, dir=1,
>>>> node=0x7eca29d4, avl=0x741f38ec)
>>>>       at ../../../../xenomai-3/include/boilerplate/avl-inner.h:395
>>>> #4  shavl_search_ge (ops=0x76f56630 <size_search_ops>, node=0x7eca29d4,
>>>> avl=0x741f38ec)
>>> The search operations parameter is last in the original code, but first
>>> in your backtrace. Does this mean that you had conflicts there and fixed
>>> them up manually? Since this is a new code, I'm unsure why you had such
>>> conflicts in the first place.
>>>
>>> Could you check with a pristine -next tree so that we may compare our
>>> results from the same code base?
>>>
>>> Thanks,
>>>
>> Philippe,
>>
>> I went back to do a full git clone, checkout, patch, rebuild, and I
>> found that when I fetched the -next branch that the patch had been
>> comitted already.  So I proceeded to skip to the build step, and then
>> retest the simultaneous execution of two instances of the smokey
>> memory_pshared test, but still got the same failure.
>>
>> Are you seeing the test failure with your build?
> No, that is why I've asked for a confirmation actually.
>
>> I've never had any reason to doubt the validity of my SDK or build
>> procedures, but if your build is passing the test, something must be
>> wrong on my end.
>>
> Not necessarily, maybe I got lucky because of some setting. I'll follow
> up on this when I have tested more configurations.
>
OK, I'll hold off on any further testing then.  My script that runs the
failing test looks like this:

ulimit -c unlimited
smokey --run=memory_pshared --verbose=2 --mem-pool-size=32M --session=foo

It ran multiple instances simultaneously just fine by omitting the
"--session=foo" switch, and with a mem-pool-size of anything above
1M.  When I ran it with --mem-pool-size=1M I got this:

# smokey --run=memory_pshared --verbose=2 --mem-pool-size=1M
== memcheck started for pshared at Thu Jan  1 03:30:37 1970

      seq_heap_size=2048k
      random_alloc_rounds=1024
      pattern_heap_size=128k
      pattern_check_rounds=128
smokey: test memory_pshared failed: Cannot allocate memory
    2"957.744| WARNING: [smokey] heapobj_init() failed for 1073352 
bytes, raise --mem-pool-size?
cannot get arena size for heap size 1048576
failed with 1024k heap, 16-byte block (pow2)

Here's some md5sums of files in case a spot-check is desired.

ba840ed4e41eace931c9f0248c987455  boilerplate/hash.c
cc5e252def896e0c004300b4f064b027  boilerplate/init/bootstrap.c
1c000d2f10dd4dfff07615de67cd193f  boilerplate/time.c
87d328c980f6a2e29f91f35d8dad99b4  boilerplate/setup.c
db1d59e47276d2f9ccf24a8697b05392  boilerplate/heapmem.c
c865d144a6b66d6c91708fbf276ae8e2  boilerplate/version.c
bcf2d4e53b87234c6cd6b28041ace6a3  boilerplate/ancillaries.c
3965251ffae674ed309404d867c9d079  boilerplate/tlsf/target.h
756dc570297f040d7bc1caec469fc096  boilerplate/tlsf/tlsf.h
1e68c9c9ca007695525e02f590f4ac93  boilerplate/tlsf/tlsf.c
cdf8be823f34a6e3e3112c024661c7c2  boilerplate/avl.c
7fc35586913521a09af968f05a34b8fe  boilerplate/debug.c
8c53c85f833fede10b8e0cb72f04269b  boilerplate/obstack.c
92a0d40e0380f9cead5a9ab87cfaa4c4  boilerplate/iniparser/dictionary.c
1b3b25f95882b4fce0d4c4eff210f372  boilerplate/iniparser/iniparser.h
83af9baca0f626da90d8a09f884a7027  boilerplate/iniparser/iniparser.c
f7c0360cf72df3002ed788d7a5ddd9b8  boilerplate/iniparser/dictionary.h
5cb8c715c7809d0b3a94a3a06925e08d  copperplate/eventobj.c
407998c2014e266566d8c05d36323998  copperplate/traceobj.c
eb01e780a57d569b7d65f73251cb3cb6  copperplate/timerobj.c
7dfc2ecd12cdfe7f4a68067b9fda8d7c  copperplate/reference.c
bf4ea39ea64567f071275450a8382bdf  copperplate/cluster.c
bfc0cad74e628595f9213283bd269b5c  copperplate/heapobj-tlsf.c
db4503787648ee50348f5dd9b945633e  copperplate/registry.c
ce88c54091eb157ab17f062df4d9ec5f  copperplate/init.c
877a34a35a46910c26c50c219967797f  copperplate/semobj.c
eecaa499b53571e59b8b67ef3f38622d  copperplate/clockobj.c
5da1685ad689ad2f155052cc69900d85  copperplate/heapobj-heapmem.c
c79587e01dec3642feb6583e534bcb4d  copperplate/syncobj.c
141b9f0db30e0f078da92d67ba94cc04  copperplate/heapobj-malloc.c
0538925a54a8eddd5640d4fba92baa32  copperplate/internal.h
83d01f4d66739d24f63a955bd8fb739b  copperplate/heapobj-pshared.c
3071943e15a939f503d426c40e3537f2  copperplate/regd/fs-cobalt.c
7509b7c943fa4cd831799e7165241da6  copperplate/regd/fs-mercury.c
038b214b224caf05ec6faf0aa60ebcca  copperplate/regd/sysregfs.h
977e6dc243748b60a743612f76744e7f  copperplate/regd/fs-common.c
16a481fe0a895ff732cae61dd9cc5912  copperplate/regd/regd.c
5f9b01f80c4edf99331accd5a6dc8cc3  copperplate/threadobj.c
34e6ccd1241a663fe54fbd03eb7d516b  copperplate/internal.c

Thanks,
Steve

Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder]

On 6/15/2018 1:26 PM, Philippe Gerum wrote:
> On 06/15/2018 07:23 PM, Steve Freyder wrote:
>> On 6/14/2018 11:52 AM, Philippe Gerum wrote:
>>> On 06/14/2018 06:41 PM, Steve Freyder wrote:
>>>> #2  0x76f418fc in shavl_search_inner (ops=0x76f56630 <size_search_ops>,
>>>> delta=0x76f418fc <add_free_range+216>, n=0x7eca29d4,
>>>>       avl=0x741f38ec) at
>>>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
>>>> #3  shavl_search_nearest (ops=0x76f56630 <size_search_ops>, dir=1,
>>>> node=0x7eca29d4, avl=0x741f38ec)
>>>>       at ../../../../xenomai-3/include/boilerplate/avl-inner.h:395
>>>> #4  shavl_search_ge (ops=0x76f56630 <size_search_ops>, node=0x7eca29d4,
>>>> avl=0x741f38ec)
>>> The search operations parameter is last in the original code, but first
>>> in your backtrace. Does this mean that you had conflicts there and fixed
>>> them up manually? Since this is a new code, I'm unsure why you had such
>>> conflicts in the first place.
>>>
>>> Could you check with a pristine -next tree so that we may compare our
>>> results from the same code base?
>>>
>>> Thanks,
>>>
>> Philippe,
>>
>> I went back to do a full git clone, checkout, patch, rebuild, and I
>> found that when I fetched the -next branch that the patch had been
>> comitted already.  So I proceeded to skip to the build step, and then
>> retest the simultaneous execution of two instances of the smokey
>> memory_pshared test, but still got the same failure.
>>
>> Are you seeing the test failure with your build?
> No, that is why I've asked for a confirmation actually.
>
>> I've never had any reason to doubt the validity of my SDK or build
>> procedures, but if your build is passing the test, something must be
>> wrong on my end.
>>
> Not necessarily, maybe I got lucky because of some setting. I'll follow
> up on this when I have tested more configurations.
>
Philippe,

I don't know if it was evident from some of the other emails we've
been exchanging, but I'm running on armv7a-neon, any possibility that
might be part of the problem?  32 vs. 64-bit maybe?