Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder <steve@freyder.net>]

On 6/12/2018 1:39 AM, Philippe Gerum wrote:
> On 06/11/2018 10:37 PM, Steve Freyder wrote:
>> On 6/8/2018 7:48 AM, Philippe Gerum wrote:
>>> On 05/20/2018 08:15 PM, Philippe Gerum wrote:
>>>> On 04/27/2018 08:23 PM, Philippe Gerum wrote:
>>>>> After a comprehensive series of tests, it is now obvious that
>>>>> heapobj-pshared has gruesome performances when releasing bursts of
>>>>> small-sized objects. Allocation of large objects is not particularly
>>>>> efficient either, although not as bad. The Cobalt core allocator is
>>>>> affected by the same bug too.
>>>>>
>>>>> Those new tests are available from the wip/heapmem branch, they are
>>>>> based on the smokey framework (e.g. smokey --run=memory_* --verbose=2).
>>>>>
>>>>> Work is ongoing into the wip/heapmem branch to fix the mess, first
>>>>> addressing the issue with the core (in-kernel) allocator,
>>>>> heapobj-pshared next. Please note that wip/heapmem is a development
>>>>> branch I'm working on for testing improvements - hopefully - to the
>>>>> memory allocators, which should not be used in production.
>>>>>
>>>>> I would strongly recommend that users running single-process
>>>>> applications - i.e. not needing the shared heap feature - make sure to
>>>>> turn off heapobj-pshared when configuring, by passing --disable-pshared
>>>>> to the configuration script (or leave the feature to the default off state).
>>>>>
>>>>> As the benchmark figures show when running 'memory_pshared', the shared
>>>> memory allocator still needs to be fixed. This is next on my list.
>>> The pshared allocator is now fixed in the next branch.
>>>
>> I tried to build with the -next branch, I'm not sure if I'm doing
>> something wrong, but I'm getting the errors below.  I thought perhaps
>> the "struct shared_heap" should have been "struct shared_heap_memory",
>> so I made that change, and that resulted in the second set of errors
>> shown below that, then I decided time to post to the group:
>>
>> Thanks,
>> Steve
>>
>> ------------------------------------------------------------------------
>>
>> make[3]: Entering directory
>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>    CC       regd.o
>>    CC       fs-common.o
>> In file included from
>> ../../../../../xenomai-3/include/cobalt/wrappers.h:21:0,
>>                   from
>> /home/sdf/xenobuild/imx-xenomai/xenomai-3/include/cobalt/time.h:26,
>>                   from
>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/sys/types.h:132,
>>                   from
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:19:
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>> â?~open_heapsâ?T:
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>> dereferencing pointer to incomplete type â?~struct shared_heapâ?T
>>     heap = container_of(obj, struct shared_heap, memspec);
>>            ^
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>> initialization from incompatible pointer type
>> [-Werror=incompatible-pointer-types]
>>     heap = container_of(obj, struct shared_heap, memspec);
>>            ^
>> In file included from
>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/string.h:32:0,
>>                   from
>> ../../../../../xenomai-3/include/copperplate/heapobj.h:24,
>>                   from
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:27:
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:35: error:
>> invalid use of undefined type â?~struct shared_heapâ?T
>>     heap = container_of(obj, struct shared_heap, memspec);
>>                                     ^
>> ------------------------------------------------------------------------
>> This is with "struct shared_heap" changed to "struct shared_heap_memory":
>>
>> make[3]: Entering directory
>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>    CC       regd.o
>>    CC       fs-common.o
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>> âopen_heapsâ:
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:249:17: error:
>> âstruct shared_heap_memoryâ has no member named âubytesâ
>>     p->used = heap->ubytes;
>>                   ^
>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:250:18: error:
>> âstruct shared_heap_memoryâ has no member named âtotalâ
>>     p->total = heap->total;
>>                    ^
>> Makefile:472: recipe for target 'fs-common.o' failed
>> make[3]: *** [fs-common.o] Error 1
>>
> This may help:
>
> diff --git a/lib/copperplate/regd/fs-common.c b/lib/copperplate/regd/fs-common.c
> index ef2e6dae9..e0bdd3d04 100644
> --- a/lib/copperplate/regd/fs-common.c
> +++ b/lib/copperplate/regd/fs-common.c
> @@ -211,8 +211,8 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>   {
>   	struct sysgroup_memspec *obj, *tmp;
>   	struct heap_data *heap_data, *p;
> +	struct shared_heap_memory *heap;
>   	struct fsobstack *o = priv;
> -	struct shared_heap *heap;
>   	int ret, count, len = 0;
>   
>   	ret = heapobj_bind_session(__copperplate_setup_data.session_label);
> @@ -244,10 +244,10 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>   	for_each_sysgroup(obj, tmp, heap) {
>   		if (p - heap_data >= count)
>   			break;
> -		heap = container_of(obj, struct shared_heap, memspec);
> +		heap = container_of(obj, struct shared_heap_memory, memspec);
>   		namecpy(p->name, heap->name);
> -		p->used = heap->ubytes;
> -		p->total = heap->total;
> +		p->used = heap->used_size;
> +		p->total = heap->usable_size;
>   		p++;
>   	}
>
Philippe,

Thanks for that update, I see that it made it into the -next branch.  
The build went fine
and I updated my SDK as well with the new heapobj headers, etc.  But now 
it appears as though
g++ is unhappy with casting between void* and heapobj*, heap_memory*, 
etc that I'm guessing has
to do with alchemy headers ref'ing the heap pointers using void *. I'm 
ignoring this for now, focusing on the apps which are in C, which is 95% 
of what I have, and will look at using
-fpermissive or #pragma's to fix the C++ issues later.

This is my first attempt with pshared since the new allocator went in.  
I've always launched
sysregd during system startup, like this:

sysregd --daemon --root /var/run/xenomai/root/opus --linger

Here's a traceback which happens with our rt_queue reader app, it only 
happens when the
session already exists, not when it's the first process (except sysregd) 
in the session:

./mqueue-reader
2165 Segmentation fault      (core dumped) ./mqueue-reader "$@"

gdb session with core file:

Core was generated by `./mqueue-readerE --mem-pool-size=32M --session=opus'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x76ee8704 in ?? ()
[Current thread is 1 (LWP 2104)]
(gdb) bt
#0  0x76ee8704 in ?? ()
#1  0x76fc38c0 in shavl_search_inner (delta=0x7ec7b9f0, n=0x7ec7ba04, 
avl=0x7445c8f4) at ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
#2  shavl_search_nearest (dir=1, node=0x7ec7ba04, avl=0x7445c8f4) at 
../../../../xenomai-3/include/boilerplate/avl-inner.h:393
#3  shavl_search_ge (node=0x7ec7ba04, avl=0x7445c8f4) at 
../../../../xenomai-3/include/boilerplate/avl-inner.h:411
#4  find_suitable_range (size=1536, ext=0x7445c8b8) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:204
#5  reserve_page_range (size=1536, ext=0x7445c8b8) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:215
#6  add_free_range (heap=0x2, heap@entry=0x7445c000, bsize=2127018500, 
bsize@entry=1536, log2size=1950728192, log2size@entry=0)
     at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:422
#7  0x76fc3b9c in sheapmem_alloc (heap=0x7445c000, size=size@entry=1272) 
at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:543
#8  0x76fc49d0 in xnmalloc (size=size@entry=1272) at 
../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:1177
#9  0x76fc18d4 in __threadobj_alloc 
(tcb_struct_size=tcb_struct_size@entry=248, 
wait_union_size=wait_union_size@entry=1024,
     thobj_offset=thobj_offset@entry=0) at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1118
#10 0x76fc27a0 in main_overlay () at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1787
#11 threadobj_pkg_init (anon_session=anon_session@entry=0) at 
../../../../xenomai-3/lib/copperplate/threadobj.c:1818
#12 0x76fbfa1c in copperplate_init () at 
../../../../xenomai-3/lib/copperplate/init.c:225
#13 0x76f9e534 in __xenomai_init (argcp=0x7ec7bbfc, 
argcp@entry=0x7ec7bc44, argvp=0x103d428, argvp@entry=0x7ec7bc40,
     me=me@entry=0x76fa6460 "program") at 
../../../../xenomai-3/lib/boilerplate/setup.c:601
#14 0x76f9ecf8 in xenomai_init (argcp=0x7ec7bc44, 
argcp@entry=0x7ec7bc3c, argvp=argvp@entry=0x7ec7bc40)
     at ../../../../xenomai-3/lib/boilerplate/setup.c:656
#15 0x00010f0c in call_init (argvp=0x7ec7bc40, argcp=0x7ec7bc3c) at 
../../../../../xenomai-3/lib/boilerplate/init/bootstrap.c:105
#16 xenomai_bootstrap () at 
../../../../../xenomai-3/lib/boilerplate/init/bootstrap.c:169
#17 0x00011864 in __libc_csu_init (argc=3, argv=0x7ec7bdd4, 
envp=0x7ec7bde4) at /usr/src/debug/glibc/2.23-r0/git/csu/elf-init.c:88
#18 0x41408088 in __libc_start_main () from /lib/libc.so.6
#19 0x00010f68 in _start () at ../sysdeps/arm/start.S:124
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

The mqueue-reader app is larger than it should be so if this traceback 
isn't enough to lead
you to a solution and you need some test code I will trim it down to 
something much smaller that
still shows the issue and send that along.

Thanks in advance,
Steve