Re: [Xenomai] Performance issue with memory allocators

[Steve Freyder <steve@freyder.net>]

On 6/13/2018 2:06 AM, Philippe Gerum wrote:
> On 06/12/2018 09:15 PM, Steve Freyder wrote:
>> On 6/12/2018 1:39 AM, Philippe Gerum wrote:
>>> On 06/11/2018 10:37 PM, Steve Freyder wrote:
>>>> On 6/8/2018 7:48 AM, Philippe Gerum wrote:
>>>>> On 05/20/2018 08:15 PM, Philippe Gerum wrote:
>>>>>> On 04/27/2018 08:23 PM, Philippe Gerum wrote:
>>>>>>> After a comprehensive series of tests, it is now obvious that
>>>>>>> heapobj-pshared has gruesome performances when releasing bursts of
>>>>>>> small-sized objects. Allocation of large objects is not particularly
>>>>>>> efficient either, although not as bad. The Cobalt core allocator is
>>>>>>> affected by the same bug too.
>>>>>>>
>>>>>>> Those new tests are available from the wip/heapmem branch, they are
>>>>>>> based on the smokey framework (e.g. smokey --run=memory_*
>>>>>>> --verbose=2).
>>>>>>>
>>>>>>> Work is ongoing into the wip/heapmem branch to fix the mess, first
>>>>>>> addressing the issue with the core (in-kernel) allocator,
>>>>>>> heapobj-pshared next. Please note that wip/heapmem is a development
>>>>>>> branch I'm working on for testing improvements - hopefully - to the
>>>>>>> memory allocators, which should not be used in production.
>>>>>>>
>>>>>>> I would strongly recommend that users running single-process
>>>>>>> applications - i.e. not needing the shared heap feature - make
>>>>>>> sure to
>>>>>>> turn off heapobj-pshared when configuring, by passing
>>>>>>> --disable-pshared
>>>>>>> to the configuration script (or leave the feature to the default
>>>>>>> off state).
>>>>>>>
>>>>>>> As the benchmark figures show when running 'memory_pshared', the
>>>>>>> shared
>>>>>> memory allocator still needs to be fixed. This is next on my list.
>>>>> The pshared allocator is now fixed in the next branch.
>>>>>
>>>> I tried to build with the -next branch, I'm not sure if I'm doing
>>>> something wrong, but I'm getting the errors below.  I thought perhaps
>>>> the "struct shared_heap" should have been "struct shared_heap_memory",
>>>> so I made that change, and that resulted in the second set of errors
>>>> shown below that, then I decided time to post to the group:
>>>>
>>>> Thanks,
>>>> Steve
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> make[3]: Entering directory
>>>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>>>     CC       regd.o
>>>>     CC       fs-common.o
>>>> In file included from
>>>> ../../../../../xenomai-3/include/cobalt/wrappers.h:21:0,
>>>>                    from
>>>> /home/sdf/xenobuild/imx-xenomai/xenomai-3/include/cobalt/time.h:26,
>>>>                    from
>>>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/sys/types.h:132,
>>>>
>>>>                    from
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:19:
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>>>> â?~open_heapsâ?T:
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>>>> dereferencing pointer to incomplete type â?~struct shared_heapâ?T
>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>             ^
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:10: error:
>>>> initialization from incompatible pointer type
>>>> [-Werror=incompatible-pointer-types]
>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>             ^
>>>> In file included from
>>>> /opt/emac/5.1/sysroots/armv7a-neon-emac-linux-gnueabi/usr/include/string.h:32:0,
>>>>
>>>>                    from
>>>> ../../../../../xenomai-3/include/copperplate/heapobj.h:24,
>>>>                    from
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:27:
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:247:35: error:
>>>> invalid use of undefined type â?~struct shared_heapâ?T
>>>>      heap = container_of(obj, struct shared_heap, memspec);
>>>>                                      ^
>>>> ------------------------------------------------------------------------
>>>> This is with "struct shared_heap" changed to "struct
>>>> shared_heap_memory":
>>>>
>>>> make[3]: Entering directory
>>>> '/home/sdf/xenobuild/imx-xenomai/build/xenomai-3/lib/copperplate/regd'
>>>>     CC       regd.o
>>>>     CC       fs-common.o
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c: In function
>>>> âopen_heapsâ:
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:249:17: error:
>>>> âstruct shared_heap_memoryâ has no member named âubytesâ
>>>>      p->used = heap->ubytes;
>>>>                    ^
>>>> ../../../../../xenomai-3/lib/copperplate/regd/fs-common.c:250:18: error:
>>>> âstruct shared_heap_memoryâ has no member named âtotalâ
>>>>      p->total = heap->total;
>>>>                     ^
>>>> Makefile:472: recipe for target 'fs-common.o' failed
>>>> make[3]: *** [fs-common.o] Error 1
>>>>
>>> This may help:
>>>
>>> diff --git a/lib/copperplate/regd/fs-common.c
>>> b/lib/copperplate/regd/fs-common.c
>>> index ef2e6dae9..e0bdd3d04 100644
>>> --- a/lib/copperplate/regd/fs-common.c
>>> +++ b/lib/copperplate/regd/fs-common.c
>>> @@ -211,8 +211,8 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>>>    {
>>>        struct sysgroup_memspec *obj, *tmp;
>>>        struct heap_data *heap_data, *p;
>>> +    struct shared_heap_memory *heap;
>>>        struct fsobstack *o = priv;
>>> -    struct shared_heap *heap;
>>>        int ret, count, len = 0;
>>>          ret =
>>> heapobj_bind_session(__copperplate_setup_data.session_label);
>>> @@ -244,10 +244,10 @@ int open_heaps(struct fsobj *fsobj, void *priv)
>>>        for_each_sysgroup(obj, tmp, heap) {
>>>            if (p - heap_data >= count)
>>>                break;
>>> -        heap = container_of(obj, struct shared_heap, memspec);
>>> +        heap = container_of(obj, struct shared_heap_memory, memspec);
>>>            namecpy(p->name, heap->name);
>>> -        p->used = heap->ubytes;
>>> -        p->total = heap->total;
>>> +        p->used = heap->used_size;
>>> +        p->total = heap->usable_size;
>>>            p++;
>>>        }
>>>
>> Philippe,
>>
>> Thanks for that update, I see that it made it into the -next branch.
>> The build went fine
>> and I updated my SDK as well with the new heapobj headers, etc.  But now
>> it appears as though
>> g++ is unhappy with casting between void* and heapobj*, heap_memory*,
>> etc that I'm guessing has
>> to do with alchemy headers ref'ing the heap pointers using void *. I'm
>> ignoring this for now, focusing on the apps which are in C, which is 95%
>> of what I have, and will look at using
>> -fpermissive or #pragma's to fix the C++ issues later.
>>
> I fixed this in the repo, only tested with a trivial C++ app pulling in
> the headers though. Please let me know if that works for you too.
>
>> This is my first attempt with pshared since the new allocator went in.
>> I've always launched
>> sysregd during system startup, like this:
>>
>> sysregd --daemon --root /var/run/xenomai/root/opus --linger
>>
>> Here's a traceback which happens with our rt_queue reader app, it only
>> happens when the
>> session already exists, not when it's the first process (except sysregd)
>> in the session:
>>
>> ./mqueue-reader
>> 2165 Segmentation fault      (core dumped) ./mqueue-reader "$@"
>>
>> gdb session with core file:
>>
>> Core was generated by `./mqueue-readerE --mem-pool-size=32M
>> --session=opus'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  0x76ee8704 in ?? ()
>> [Current thread is 1 (LWP 2104)]
>> (gdb) bt
>> #0  0x76ee8704 in ?? ()
>> #1  0x76fc38c0 in shavl_search_inner (delta=0x7ec7b9f0, n=0x7ec7ba04,
>> avl=0x7445c8f4) at
>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:285
>> #2  shavl_search_nearest (dir=1, node=0x7ec7ba04, avl=0x7445c8f4) at
>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:393
>> #3  shavl_search_ge (node=0x7ec7ba04, avl=0x7445c8f4) at
>> ../../../../xenomai-3/include/boilerplate/avl-inner.h:411
>> #4  find_suitable_range (size=1536, ext=0x7445c8b8) at
>> ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:204
>> #5  reserve_page_range (size=1536, ext=0x7445c8b8) at
>> ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:215
>> #6  add_free_range (heap=0x2, heap@entry=0x7445c000, bsize=2127018500,
>> bsize@entry=1536, log2size=1950728192, log2size@entry=0)
>>      at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:422
>> #7  0x76fc3b9c in sheapmem_alloc (heap=0x7445c000, size=size@entry=1272)
>> at ../../../../xenomai-3/lib/copperplate/heapobj-pshared.c:543
>>
> Going from heap=0x7445c000 with size=1272 to heap=0x2 with
> size=2127018500 looks weird, and won't work obviously. Any chance a
> stack overflow would be at work here?
>
>> The mqueue-reader app is larger than it should be so if this traceback
>> isn't enough to lead
>> you to a solution and you need some test code I will trim it down to
>> something much smaller that
>> still shows the issue and send that along.
>>
> You may want to run the new stress test exercising all of the memory
> allocators, specifically for the heapmem-pshared:
>
> $ smokey --run=memory_pshared --verbose=2
>
> If that one runs properly on your setup, I would suspect an issue local
> to the app (e.g. such as a stack overflow).
>
Philippe,

First, thanks for the update on the g++ header issue, my compiles are 
all clean now.

I rebuilt my program with a 512K stack, same problem with the SEGV.

Tried something different, without running my application at all, I ran 
this script:

ulimit -c unlimited
smokey --run=memory_pshared --verbose=2 --mem-pool-size=32M --session=foo

it ran fine, but if I ran it a second time, without waiting for the 
first invocation to
finish, then I got the same SEGV that I previously was seeing with my 
program and the
traceback looks the same.

It's as if there's an initialization issue when you're not the creator 
of the
pshared heap.