[Xenomai] Segfaults and ENOMEM during rt_event_create()

[Xenomai] Segfaults and ENOMEM during rt_event_create()

[Charles Kiorpes]

Hello,

I am trying to port a fairly large application from Xenomai 2.6.4 to
Xenomai 3.0-rc6

Most of the port has been decently painless, but there is one issue I
can't seem to get past.

I am consistently getting ENOMEM errors and segfaults in
libcopperplate.so during rt_event_create(), and always at the same
spot in my application.

If a segfault occurrs, the dmesg output from the segfault looks like this:
[   235.422300] PortProcess[1502]: segfault at f1b3ebeb ip b76c14b7 sp
bfeae498 error 5 in libcopperplate.so.0.0.0[b76b7000+14000]

The application uses a large number of named event flag groups.  At
the time that the error occurs, it is looping to create 32 new named
event groups.

I have increased these values in my kernel config:
CONFIG_XENO_OPT_REGISTRY_NRSLOTS=4096
CONFIG_XENO_OPT_SYS_HEAPSZ=512
CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
CONFIG_XENO_OPT_SHARED_HEAPSZ=128

When I dig through the /proc/xenomai filesystem, I see that:
- I am only using 383/4096 registry slots
- My system heap is about half full (284160KB free of 524288KB)
- My shared and private heaps are basically empty

My application is launched with --mem-pool-size=400000.  The main heap
(in /registry/user/session/system) has about 220KB used at the point
that the error occurs.
Increasing the mem-pool-size does not change the behavior.  I have
also tried further increasing the values of my system (1024), private
(256), and shared (256) heapsizes, with no effect.

Obviously I've missed something, but I can't figure out where the
error is occurring.

System details:
OS: Debian Jessie 32-bit
Kernel: 3.18.12 - patched using Xenomai 3.0-rc6
System: Intel Xeon W3503 (x86 architecture)
I-Pipe version 1

Thanks for your help!

- Charles Kiorpes

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 05/08/2015 15:09, Charles Kiorpes wrote:
> Hello,
> 
> I am trying to port a fairly large application from Xenomai 2.6.4 to
> Xenomai 3.0-rc6
> 
> Most of the port has been decently painless, but there is one issue I
> can't seem to get past.
> 
> I am consistently getting ENOMEM errors and segfaults in
> libcopperplate.so during rt_event_create(), and always at the same
> spot in my application.
> 
> If a segfault occurrs, the dmesg output from the segfault looks like this:
> [   235.422300] PortProcess[1502]: segfault at f1b3ebeb ip b76c14b7 sp
> bfeae498 error 5 in libcopperplate.so.0.0.0[b76b7000+14000]
> 
> The application uses a large number of named event flag groups.  At
> the time that the error occurs, it is looping to create 32 new named
> event groups.
> 
> I have increased these values in my kernel config:
> CONFIG_XENO_OPT_REGISTRY_NRSLOTS=4096
> CONFIG_XENO_OPT_SYS_HEAPSZ=512
> CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
> CONFIG_XENO_OPT_SHARED_HEAPSZ=128
> 
> When I dig through the /proc/xenomai filesystem, I see that:
> - I am only using 383/4096 registry slots
> - My system heap is about half full (284160KB free of 524288KB)
> - My shared and private heaps are basically empty
> 
> My application is launched with --mem-pool-size=400000.  The main heap
> (in /registry/user/session/system) has about 220KB used at the point
> that the error occurs.
> Increasing the mem-pool-size does not change the behavior.  I have
> also tried further increasing the values of my system (1024), private
> (256), and shared (256) heapsizes, with no effect.
> 
> Obviously I've missed something, but I can't figure out where the
> error is occurring.
> 
> System details:
> OS: Debian Jessie 32-bit
> Kernel: 3.18.12 - patched using Xenomai 3.0-rc6
> System: Intel Xeon W3503 (x86 architecture)
> I-Pipe version 1
> 

Any improvement with this patch in?

commit 574c8e2edb99ff523e765cdb8749dd5259784627
Author: Philippe Gerum <rpm@xenomai.org>
Date:   Wed Aug 5 16:08:23 2015 +0200

    lib/cobalt: fix private event state accessor

diff --git a/lib/cobalt/internal.c b/lib/cobalt/internal.c
index 348ca14..62ecb4a 100644
--- a/lib/cobalt/internal.c
+++ b/lib/cobalt/internal.c
@@ -426,7 +426,7 @@ struct cobalt_event_state
*get_event_state(cobalt_event_t *event)
 {
 	return event->flags & COBALT_EVENT_SHARED ?
 		cobalt_umm_shared + event->state_offset :
-		cobalt_umm_shared + event->state_offset;
+		cobalt_umm_private + event->state_offset;
 }

 int cobalt_event_init(cobalt_event_t *event, unsigned int value,



-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Charles Kiorpes]

Sorry if this gets to you twice, I forgot to "Reply to all"

After applying the patch and rebuilding, my application still
segfaults at the same point, with the same error in my dmesg output.

For a little more info, here are the config flags I'm using to build
the libraries:
--with-core=cobalt
--disable-debug
--enable-pshared
--enable-registry=/home/ckiorpes/xen-userspace/
--enable-tls
--enable-smp

The issue occurs whether or not I disable the registry with the
--no-registry flag.


-Charles

On Wed, Aug 5, 2015 at 11:42 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> On 05/08/2015 15:09, Charles Kiorpes wrote:
>> Hello,
>>
>> I am trying to port a fairly large application from Xenomai 2.6.4 to
>> Xenomai 3.0-rc6
>>
>> Most of the port has been decently painless, but there is one issue I
>> can't seem to get past.
>>
>> I am consistently getting ENOMEM errors and segfaults in
>> libcopperplate.so during rt_event_create(), and always at the same
>> spot in my application.
>>
>> If a segfault occurrs, the dmesg output from the segfault looks like this:
>> [   235.422300] PortProcess[1502]: segfault at f1b3ebeb ip b76c14b7 sp
>> bfeae498 error 5 in libcopperplate.so.0.0.0[b76b7000+14000]
>>
>> The application uses a large number of named event flag groups.  At
>> the time that the error occurs, it is looping to create 32 new named
>> event groups.
>>
>> I have increased these values in my kernel config:
>> CONFIG_XENO_OPT_REGISTRY_NRSLOTS=4096
>> CONFIG_XENO_OPT_SYS_HEAPSZ=512
>> CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
>> CONFIG_XENO_OPT_SHARED_HEAPSZ=128
>>
>> When I dig through the /proc/xenomai filesystem, I see that:
>> - I am only using 383/4096 registry slots
>> - My system heap is about half full (284160KB free of 524288KB)
>> - My shared and private heaps are basically empty
>>
>> My application is launched with --mem-pool-size=400000.  The main heap
>> (in /registry/user/session/system) has about 220KB used at the point
>> that the error occurs.
>> Increasing the mem-pool-size does not change the behavior.  I have
>> also tried further increasing the values of my system (1024), private
>> (256), and shared (256) heapsizes, with no effect.
>>
>> Obviously I've missed something, but I can't figure out where the
>> error is occurring.
>>
>> System details:
>> OS: Debian Jessie 32-bit
>> Kernel: 3.18.12 - patched using Xenomai 3.0-rc6
>> System: Intel Xeon W3503 (x86 architecture)
>> I-Pipe version 1
>>
>
> Any improvement with this patch in?
>
> commit 574c8e2edb99ff523e765cdb8749dd5259784627
> Author: Philippe Gerum <rpm@xenomai.org>
> Date:   Wed Aug 5 16:08:23 2015 +0200
>
>     lib/cobalt: fix private event state accessor
>
> diff --git a/lib/cobalt/internal.c b/lib/cobalt/internal.c
> index 348ca14..62ecb4a 100644
> --- a/lib/cobalt/internal.c
> +++ b/lib/cobalt/internal.c
> @@ -426,7 +426,7 @@ struct cobalt_event_state
> *get_event_state(cobalt_event_t *event)
>  {
>         return event->flags & COBALT_EVENT_SHARED ?
>                 cobalt_umm_shared + event->state_offset :
> -               cobalt_umm_shared + event->state_offset;
> +               cobalt_umm_private + event->state_offset;
>  }
>
>  int cobalt_event_init(cobalt_event_t *event, unsigned int value,
>
>
>
> --
> Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Gilles Chanteperdrix]

On Wed, Aug 05, 2015 at 05:42:37PM +0200, Philippe Gerum wrote:
> On 05/08/2015 15:09, Charles Kiorpes wrote:
> > Hello,
> > 
> > I am trying to port a fairly large application from Xenomai 2.6.4 to
> > Xenomai 3.0-rc6
> > 
> > Most of the port has been decently painless, but there is one issue I
> > can't seem to get past.
> > 
> > I am consistently getting ENOMEM errors and segfaults in
> > libcopperplate.so during rt_event_create(), and always at the same
> > spot in my application.
> > 
> > If a segfault occurrs, the dmesg output from the segfault looks like this:
> > [   235.422300] PortProcess[1502]: segfault at f1b3ebeb ip b76c14b7 sp
> > bfeae498 error 5 in libcopperplate.so.0.0.0[b76b7000+14000]
> > 
> > The application uses a large number of named event flag groups.  At
> > the time that the error occurs, it is looping to create 32 new named
> > event groups.
> > 
> > I have increased these values in my kernel config:
> > CONFIG_XENO_OPT_REGISTRY_NRSLOTS=4096
> > CONFIG_XENO_OPT_SYS_HEAPSZ=512
> > CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
> > CONFIG_XENO_OPT_SHARED_HEAPSZ=128
> > 
> > When I dig through the /proc/xenomai filesystem, I see that:
> > - I am only using 383/4096 registry slots
> > - My system heap is about half full (284160KB free of 524288KB)
> > - My shared and private heaps are basically empty
> > 
> > My application is launched with --mem-pool-size=400000.  The main heap
> > (in /registry/user/session/system) has about 220KB used at the point
> > that the error occurs.
> > Increasing the mem-pool-size does not change the behavior.  I have
> > also tried further increasing the values of my system (1024), private
> > (256), and shared (256) heapsizes, with no effect.
> > 
> > Obviously I've missed something, but I can't figure out where the
> > error is occurring.
> > 
> > System details:
> > OS: Debian Jessie 32-bit
> > Kernel: 3.18.12 - patched using Xenomai 3.0-rc6
> > System: Intel Xeon W3503 (x86 architecture)
> > I-Pipe version 1
> > 
> 
> Any improvement with this patch in?
> 
> commit 574c8e2edb99ff523e765cdb8749dd5259784627
> Author: Philippe Gerum <rpm@xenomai.org>
> Date:   Wed Aug 5 16:08:23 2015 +0200
> 
>     lib/cobalt: fix private event state accessor
> 
> diff --git a/lib/cobalt/internal.c b/lib/cobalt/internal.c
> index 348ca14..62ecb4a 100644
> --- a/lib/cobalt/internal.c
> +++ b/lib/cobalt/internal.c
> @@ -426,7 +426,7 @@ struct cobalt_event_state
> *get_event_state(cobalt_event_t *event)
>  {
>  	return event->flags & COBALT_EVENT_SHARED ?

I would put parens here.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/05/2015 07:24 PM, Gilles Chanteperdrix wrote:
> On Wed, Aug 05, 2015 at 05:42:37PM +0200, Philippe Gerum wrote:
>> On 05/08/2015 15:09, Charles Kiorpes wrote:
>>> Hello,
>>>
>>> I am trying to port a fairly large application from Xenomai 2.6.4 to
>>> Xenomai 3.0-rc6
>>>
>>> Most of the port has been decently painless, but there is one issue I
>>> can't seem to get past.
>>>
>>> I am consistently getting ENOMEM errors and segfaults in
>>> libcopperplate.so during rt_event_create(), and always at the same
>>> spot in my application.
>>>
>>> If a segfault occurrs, the dmesg output from the segfault looks like this:
>>> [   235.422300] PortProcess[1502]: segfault at f1b3ebeb ip b76c14b7 sp
>>> bfeae498 error 5 in libcopperplate.so.0.0.0[b76b7000+14000]
>>>
>>> The application uses a large number of named event flag groups.  At
>>> the time that the error occurs, it is looping to create 32 new named
>>> event groups.
>>>
>>> I have increased these values in my kernel config:
>>> CONFIG_XENO_OPT_REGISTRY_NRSLOTS=4096
>>> CONFIG_XENO_OPT_SYS_HEAPSZ=512
>>> CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
>>> CONFIG_XENO_OPT_SHARED_HEAPSZ=128
>>>
>>> When I dig through the /proc/xenomai filesystem, I see that:
>>> - I am only using 383/4096 registry slots
>>> - My system heap is about half full (284160KB free of 524288KB)
>>> - My shared and private heaps are basically empty
>>>
>>> My application is launched with --mem-pool-size=400000.  The main heap
>>> (in /registry/user/session/system) has about 220KB used at the point
>>> that the error occurs.
>>> Increasing the mem-pool-size does not change the behavior.  I have
>>> also tried further increasing the values of my system (1024), private
>>> (256), and shared (256) heapsizes, with no effect.
>>>
>>> Obviously I've missed something, but I can't figure out where the
>>> error is occurring.
>>>
>>> System details:
>>> OS: Debian Jessie 32-bit
>>> Kernel: 3.18.12 - patched using Xenomai 3.0-rc6
>>> System: Intel Xeon W3503 (x86 architecture)
>>> I-Pipe version 1
>>>
>>
>> Any improvement with this patch in?
>>
>> commit 574c8e2edb99ff523e765cdb8749dd5259784627
>> Author: Philippe Gerum <rpm@xenomai.org>
>> Date:   Wed Aug 5 16:08:23 2015 +0200
>>
>>     lib/cobalt: fix private event state accessor
>>
>> diff --git a/lib/cobalt/internal.c b/lib/cobalt/internal.c
>> index 348ca14..62ecb4a 100644
>> --- a/lib/cobalt/internal.c
>> +++ b/lib/cobalt/internal.c
>> @@ -426,7 +426,7 @@ struct cobalt_event_state
>> *get_event_state(cobalt_event_t *event)
>>  {
>>  	return event->flags & COBALT_EVENT_SHARED ?
> 
> I would put parens here.
> 

I'm trying hard, but I don't see why. The standard is pretty clear about
the ternary conditional (§6.5.15): its precedence is right above the
assignment operators and below anything else, expr0 is evaluated first
and followed by a sequence point.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/05/2015 06:34 PM, Charles Kiorpes wrote:
> Sorry if this gets to you twice, I forgot to "Reply to all"
> 
> After applying the patch and rebuilding, my application still
> segfaults at the same point, with the same error in my dmesg output.
> 
> For a little more info, here are the config flags I'm using to build
> the libraries:
> --with-core=cobalt
> --disable-debug
> --enable-pshared

Ok, so if pshared was enabled, the bug fixed by this patch could not
bite anyway.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Gilles Chanteperdrix]

On Thu, Aug 06, 2015 at 09:40:53AM +0200, Philippe Gerum wrote:
> On 08/05/2015 07:24 PM, Gilles Chanteperdrix wrote:
> > On Wed, Aug 05, 2015 at 05:42:37PM +0200, Philippe Gerum wrote:
> >> On 05/08/2015 15:09, Charles Kiorpes wrote:
> >>> Hello,
> >>>
> >>> I am trying to port a fairly large application from Xenomai 2.6.4 to
> >>> Xenomai 3.0-rc6
> >>>
> >>> Most of the port has been decently painless, but there is one issue I
> >>> can't seem to get past.
> >>>
> >>> I am consistently getting ENOMEM errors and segfaults in
> >>> libcopperplate.so during rt_event_create(), and always at the same
> >>> spot in my application.
> >>>
> >>> If a segfault occurrs, the dmesg output from the segfault looks like this:
> >>> [   235.422300] PortProcess[1502]: segfault at f1b3ebeb ip b76c14b7 sp
> >>> bfeae498 error 5 in libcopperplate.so.0.0.0[b76b7000+14000]
> >>>
> >>> The application uses a large number of named event flag groups.  At
> >>> the time that the error occurs, it is looping to create 32 new named
> >>> event groups.
> >>>
> >>> I have increased these values in my kernel config:
> >>> CONFIG_XENO_OPT_REGISTRY_NRSLOTS=4096
> >>> CONFIG_XENO_OPT_SYS_HEAPSZ=512
> >>> CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
> >>> CONFIG_XENO_OPT_SHARED_HEAPSZ=128
> >>>
> >>> When I dig through the /proc/xenomai filesystem, I see that:
> >>> - I am only using 383/4096 registry slots
> >>> - My system heap is about half full (284160KB free of 524288KB)
> >>> - My shared and private heaps are basically empty
> >>>
> >>> My application is launched with --mem-pool-size=400000.  The main heap
> >>> (in /registry/user/session/system) has about 220KB used at the point
> >>> that the error occurs.
> >>> Increasing the mem-pool-size does not change the behavior.  I have
> >>> also tried further increasing the values of my system (1024), private
> >>> (256), and shared (256) heapsizes, with no effect.
> >>>
> >>> Obviously I've missed something, but I can't figure out where the
> >>> error is occurring.
> >>>
> >>> System details:
> >>> OS: Debian Jessie 32-bit
> >>> Kernel: 3.18.12 - patched using Xenomai 3.0-rc6
> >>> System: Intel Xeon W3503 (x86 architecture)
> >>> I-Pipe version 1
> >>>
> >>
> >> Any improvement with this patch in?
> >>
> >> commit 574c8e2edb99ff523e765cdb8749dd5259784627
> >> Author: Philippe Gerum <rpm@xenomai.org>
> >> Date:   Wed Aug 5 16:08:23 2015 +0200
> >>
> >>     lib/cobalt: fix private event state accessor
> >>
> >> diff --git a/lib/cobalt/internal.c b/lib/cobalt/internal.c
> >> index 348ca14..62ecb4a 100644
> >> --- a/lib/cobalt/internal.c
> >> +++ b/lib/cobalt/internal.c
> >> @@ -426,7 +426,7 @@ struct cobalt_event_state
> >> *get_event_state(cobalt_event_t *event)
> >>  {
> >>  	return event->flags & COBALT_EVENT_SHARED ?
> > 
> > I would put parens here.
> > 
> 
> I'm trying hard, but I don't see why. The standard is pretty clear about
> the ternary conditional (§6.5.15): its precedence is right above the
> assignment operators and below anything else, expr0 is evaluated first
> and followed by a sequence point.

I had trouble with if (a & b && c), so am now putting parens around
(a & b) by principle, even if not needed.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/06/2015 09:49 AM, Gilles Chanteperdrix wrote:
> On Thu, Aug 06, 2015 at 09:40:53AM +0200, Philippe Gerum wrote:
>> On 08/05/2015 07:24 PM, Gilles Chanteperdrix wrote:
>>> On Wed, Aug 05, 2015 at 05:42:37PM +0200, Philippe Gerum wrote:
>>>> On 05/08/2015 15:09, Charles Kiorpes wrote:
>>>>> Hello,
>>>>>
>>>>> I am trying to port a fairly large application from Xenomai 2.6.4 to
>>>>> Xenomai 3.0-rc6
>>>>>
>>>>> Most of the port has been decently painless, but there is one issue I
>>>>> can't seem to get past.
>>>>>
>>>>> I am consistently getting ENOMEM errors and segfaults in
>>>>> libcopperplate.so during rt_event_create(), and always at the same
>>>>> spot in my application.
>>>>>
>>>>> If a segfault occurrs, the dmesg output from the segfault looks like this:
>>>>> [   235.422300] PortProcess[1502]: segfault at f1b3ebeb ip b76c14b7 sp
>>>>> bfeae498 error 5 in libcopperplate.so.0.0.0[b76b7000+14000]
>>>>>
>>>>> The application uses a large number of named event flag groups.  At
>>>>> the time that the error occurs, it is looping to create 32 new named
>>>>> event groups.
>>>>>
>>>>> I have increased these values in my kernel config:
>>>>> CONFIG_XENO_OPT_REGISTRY_NRSLOTS=4096
>>>>> CONFIG_XENO_OPT_SYS_HEAPSZ=512
>>>>> CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
>>>>> CONFIG_XENO_OPT_SHARED_HEAPSZ=128
>>>>>
>>>>> When I dig through the /proc/xenomai filesystem, I see that:
>>>>> - I am only using 383/4096 registry slots
>>>>> - My system heap is about half full (284160KB free of 524288KB)
>>>>> - My shared and private heaps are basically empty
>>>>>
>>>>> My application is launched with --mem-pool-size=400000.  The main heap
>>>>> (in /registry/user/session/system) has about 220KB used at the point
>>>>> that the error occurs.
>>>>> Increasing the mem-pool-size does not change the behavior.  I have
>>>>> also tried further increasing the values of my system (1024), private
>>>>> (256), and shared (256) heapsizes, with no effect.
>>>>>
>>>>> Obviously I've missed something, but I can't figure out where the
>>>>> error is occurring.
>>>>>
>>>>> System details:
>>>>> OS: Debian Jessie 32-bit
>>>>> Kernel: 3.18.12 - patched using Xenomai 3.0-rc6
>>>>> System: Intel Xeon W3503 (x86 architecture)
>>>>> I-Pipe version 1
>>>>>
>>>>
>>>> Any improvement with this patch in?
>>>>
>>>> commit 574c8e2edb99ff523e765cdb8749dd5259784627
>>>> Author: Philippe Gerum <rpm@xenomai.org>
>>>> Date:   Wed Aug 5 16:08:23 2015 +0200
>>>>
>>>>     lib/cobalt: fix private event state accessor
>>>>
>>>> diff --git a/lib/cobalt/internal.c b/lib/cobalt/internal.c
>>>> index 348ca14..62ecb4a 100644
>>>> --- a/lib/cobalt/internal.c
>>>> +++ b/lib/cobalt/internal.c
>>>> @@ -426,7 +426,7 @@ struct cobalt_event_state
>>>> *get_event_state(cobalt_event_t *event)
>>>>  {
>>>>  	return event->flags & COBALT_EVENT_SHARED ?
>>>
>>> I would put parens here.
>>>
>>
>> I'm trying hard, but I don't see why. The standard is pretty clear about
>> the ternary conditional (§6.5.15): its precedence is right above the
>> assignment operators and below anything else, expr0 is evaluated first
>> and followed by a sequence point.
> 
> I had trouble with if (a & b && c), so am now putting parens around
> (a & b) by principle, even if not needed.
> 

I agree in general, but the ternary operator is a bit special since by
design, it won't compete for precedence with the expressions it evaluates.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[dietmar.schindler]

> Von: Philippe Gerum
> Gesendet: Donnerstag, 6. August 2015 10:01
>
> ..., but the ternary operator is a bit special since by
> design, it won't compete for precedence with the expressions it evaluates.

Actually, this is not special - no operator will "compete for precedence with the expressions it evaluates", because _which_ expressions it evaluates is determined by the very outcome of the "competition for precedence". And sure enough, the expression

    b = a ? 0 : 1

is not equivalent to

    (b = a) ? 0 : 1

(but by this remark I do not at all want to advocate the undue use of parentheses).

--
Best regards,
Dietmar Schindler
________________________________
manroland web systems GmbH -- Managing Director: Joern Gosse
Registered Office: Augsburg -- Trade Register: AG Augsburg -- HRB-No.: 26816 -- VAT: DE281389840

Confidentiality note:
This eMail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient, you are hereby notified that any use or dissemination of this communication is strictly prohibited. If you have received this eMail in error, then please delete this eMail.

! Please consider your environmental responsibility before printing this eMail !
________________________________

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Charles Kiorpes]

> Could you --enable-debug, and have gdb determine the offending
> instruction? TIA,

The gdb output is as follows:

Program received signal SIGSEGV, Segmentation fault.
get_free_range (heap=heap@entry=0x6dc07000, bsize=bsize@entry=256, log2size=8)
    at ../../../xenomai-3.0-rc6/lib/copperplate/heapobj-pshared.c:242
242                                           freepage =
__shref_check(heap, *((memoff_t *)freepage));

Thanks,
- Charles

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/06/2015 11:25 AM, dietmar.schindler@manroland-web.com wrote:
>> Von: Philippe Gerum
>> Gesendet: Donnerstag, 6. August 2015 10:01
>>
>> ..., but the ternary operator is a bit special since by
>> design, it won't compete for precedence with the expressions it evaluates.
> 
> Actually, this is not special - no operator will "compete for precedence with the expressions it evaluates", because _which_ expressions it evaluates is determined by the very outcome of the "competition for precedence". And sure enough, the expression
> 
>     b = a ? 0 : 1
> 

I would tend to think that associativity is part of the issue. e.g.
right association solves the ambiguity between multiple assignments
operators with the same precedence.

> is not equivalent to
> 
>     (b = a) ? 0 : 1
> 
> (but by this remark I do not at all want to advocate the undue use of parentheses).
> 

Clearly. This said, the original remark was rather about the precedence
between operators from the test expression and the operators forming the
ternary conditional itself.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/06/2015 05:00 PM, Charles Kiorpes wrote:
>> Could you --enable-debug, and have gdb determine the offending
>> instruction? TIA,
> 
> The gdb output is as follows:
> 
> Program received signal SIGSEGV, Segmentation fault.
> get_free_range (heap=heap@entry=0x6dc07000, bsize=bsize@entry=256, log2size=8)
>     at ../../../xenomai-3.0-rc6/lib/copperplate/heapobj-pshared.c:242
> 242                                           freepage =
> __shref_check(heap, *((memoff_t *)freepage));
> 
> Thanks,
> - Charles
> 

This may be a memory corruption issue, some metadata from the shared
heap got trashed earlier in the init sequence.

Would it be practical to switch this app to the mercury core (no need
for preempt-rt, just a plain regular desktop kernel), only for the
purpose of running the init phase over valgrind? That would entail
stubbing/bypassing the hardware-related inits (e.g. drivers) if ever
possible.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/06/2015 05:23 PM, Philippe Gerum wrote:
> On 08/06/2015 05:00 PM, Charles Kiorpes wrote:
>>> Could you --enable-debug, and have gdb determine the offending
>>> instruction? TIA,
>>
>> The gdb output is as follows:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> get_free_range (heap=heap@entry=0x6dc07000, bsize=bsize@entry=256, log2size=8)
>>     at ../../../xenomai-3.0-rc6/lib/copperplate/heapobj-pshared.c:242
>> 242                                           freepage =
>> __shref_check(heap, *((memoff_t *)freepage));
>>
>> Thanks,
>> - Charles
>>
> 
> This may be a memory corruption issue, some metadata from the shared
> heap got trashed earlier in the init sequence.
> 
> Would it be practical to switch this app to the mercury core (no need
> for preempt-rt, just a plain regular desktop kernel), only for the
> purpose of running the init phase over valgrind? That would entail
> stubbing/bypassing the hardware-related inits (e.g. drivers) if ever
> possible.
> 

Also, does this bug happen with --disable-pshared?

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Charles Kiorpes]

> This may be a memory corruption issue, some metadata from the shared
> heap got trashed earlier in the init sequence.
>
> Would it be practical to switch this app to the mercury core (no need
> for preempt-rt, just a plain regular desktop kernel), only for the
> purpose of running the init phase over valgrind? That would entail
> stubbing/bypassing the hardware-related inits (e.g. drivers) if ever
> possible.

I reconfigured to use Mercury, but I can't reproduce the error in this
configuration.  There is no longer any segfault, or ENOMEM.

Despite being unable to reproduce the problem, Valgrind output
indicates something funny happening when I call rt_event_create() on
an object that already exists.

Here's the output from Valgrind of what occurs when I call
rt_event_create() in pshared mode with an event that already exists
(I've truncated some of the call stacks, this is all from one call to
rt_event_create through a wrapper layer):

==6204== Thread 36 Task32:
==6204== Invalid read of size 1
==6204==    at 0x40E1F10: __hash_key (hash.c:66)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a2d is 5 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F14: __hash_key (hash.c:66)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a2e is 6 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F20: __hash_key (hash.c:66)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a2c is 4 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F26: __hash_key (hash.c:66)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a2f is 7 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F2F: __hash_key (hash.c:67)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a31 is 9 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F35: __hash_key (hash.c:67)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a32 is 10 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F41: __hash_key (hash.c:67)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
...
==6204==    by 0x80B87BC: SMCCommandHandler (AswLibSv.c:230)
==6204==  Address 0x4434a30 is 8 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F47: __hash_key (hash.c:67)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a33 is 11 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F50: __hash_key (hash.c:65)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a29 is 1 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F56: __hash_key (hash.c:65)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a2a is 2 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F62: __hash_key (hash.c:65)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a28 is 0 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E1F67: __hash_key (hash.c:65)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a2b is 3 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E2044: __hash_key (hash.c:84)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a42 is 26 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E204D: __hash_key (hash.c:85)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a41 is 25 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid read of size 1
==6204==    at 0x40E2056: __hash_key (hash.c:86)
==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a40 is 24 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==
==6204== Invalid free() / delete / delete[] / realloc()
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x4056C93: event_finalize (event.c:116)
==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
==6204==    by 0x4057017: rt_event_create (event.c:218)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==  Address 0x4434a28 is 0 bytes inside a block of size 28 free'd
==6204==    at 0x402A3A8: free (vg_replace_malloc.c:473)
==6204==    by 0x4073A1E: pvfree (heapobj.h:159)
==6204==    by 0x4073A1E: registry_destroy_file (registry.c:265)
==6204==    by 0x405700D: rt_event_create (event.c:217)
==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)
...
==6204==    by 0x8182909: TaskThread (TaskEngine.c:151)
==6204==    by 0x405A083: task_entry (task.c:237)
==6204==    by 0x406AEC2: thread_trampoline (internal.c:233)
==6204==    by 0x4086EFA: start_thread (pthread_create.c:309)
==6204==    by 0x432962D: clone (clone.S:129)
==6204==

Sorry for the long output.

This can be worked around by attempting an rt_event_bind() first and
only attempting to create the event if bind fails.

The above is the only complaint from Valgrind.

>Also, does this bug happen with --disable-pshared?

I am unable to run this application without pshared, because there are
two processes that need to signal each other using real-time events.

Thanks,
Charles

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/10/2015 03:52 PM, Charles Kiorpes wrote:
>> This may be a memory corruption issue, some metadata from the shared
>> heap got trashed earlier in the init sequence.
>>
>> Would it be practical to switch this app to the mercury core (no need
>> for preempt-rt, just a plain regular desktop kernel), only for the
>> purpose of running the init phase over valgrind? That would entail
>> stubbing/bypassing the hardware-related inits (e.g. drivers) if ever
>> possible.
> 
> I reconfigured to use Mercury, but I can't reproduce the error in this
> configuration.  There is no longer any segfault, or ENOMEM.
> 
> Despite being unable to reproduce the problem, Valgrind output
> indicates something funny happening when I call rt_event_create() on
> an object that already exists.
> 
> Here's the output from Valgrind of what occurs when I call
> rt_event_create() in pshared mode with an event that already exists
> (I've truncated some of the call stacks, this is all from one call to
> rt_event_create through a wrapper layer):
> 

[snip]

> This can be worked around by attempting an rt_event_bind() first and
> only attempting to create the event if bind fails.
> 

Is this case (trying to create an event flag with a conflicting name)
part of the original scenario with Cobalt?

> The above is the only complaint from Valgrind.
> 
>> Also, does this bug happen with --disable-pshared?
> 
> I am unable to run this application without pshared, because there are
> two processes that need to signal each other using real-time events.

Ok.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Charles Kiorpes]

> Is this case (trying to create an event flag with a conflicting name)
> part of the original scenario with Cobalt?

This rt_event_create() with existing name occurs a little earlier in
my application's startup phase than where the segfault occurs using
Cobalt.  However, applying the rt_event_bind() workaround and
switching back to Cobalt does not change the segfault behavior (it
still occurs at the same place), so I'm not convinced that they're
related.

When I run the processes through Valgrind using the Mercury core and
checking event existence with rt_event_bind() before attempting to
create, Valgrind finds no issues.  The application hangs at a
different location (creation of a periodic task; I'm fairly certain
that I just need to rework that section of code a little).

However, my application really need the tight timings that Cobalt
provides, and this has to eventually run on an embedded system, so
switching to Mercury is not a viable solution.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/10/2015 05:17 PM, Charles Kiorpes wrote:
>> Is this case (trying to create an event flag with a conflicting name)
>> part of the original scenario with Cobalt?
> 
> This rt_event_create() with existing name occurs a little earlier in
> my application's startup phase than where the segfault occurs using
> Cobalt.  However, applying the rt_event_bind() workaround and
> switching back to Cobalt does not change the segfault behavior (it
> still occurs at the same place), so I'm not convinced that they're
> related.
>

Ok.

> When I run the processes through Valgrind using the Mercury core and
> checking event existence with rt_event_bind() before attempting to
> create, Valgrind finds no issues.  The application hangs at a
> different location (creation of a periodic task; I'm fairly certain
> that I just need to rework that section of code a little).
> 
> However, my application really need the tight timings that Cobalt
> provides, and this has to eventually run on an embedded system, so
> switching to Mercury is not a viable solution.
> 

Understood. This switch is only mentioned as a way to enable Valgrind on
the code temporarily for debugging purpose, since this is not possible
to run Cobalt apps over it.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/10/2015 03:52 PM, Charles Kiorpes wrote:
>> This may be a memory corruption issue, some metadata from the shared
>> heap got trashed earlier in the init sequence.
>>
>> Would it be practical to switch this app to the mercury core (no need
>> for preempt-rt, just a plain regular desktop kernel), only for the
>> purpose of running the init phase over valgrind? That would entail
>> stubbing/bypassing the hardware-related inits (e.g. drivers) if ever
>> possible.
> 
> I reconfigured to use Mercury, but I can't reproduce the error in this
> configuration.  There is no longer any segfault, or ENOMEM.
> 
> Despite being unable to reproduce the problem, Valgrind output
> indicates something funny happening when I call rt_event_create() on
> an object that already exists.
> 
> Here's the output from Valgrind of what occurs when I call
> rt_event_create() in pshared mode with an event that already exists
> (I've truncated some of the call stacks, this is all from one call to
> rt_event_create through a wrapper layer):
> 
> ==6204== Thread 36 Task32:
> ==6204== Invalid read of size 1
> ==6204==    at 0x40E1F10: __hash_key (hash.c:66)
> ==6204==    by 0x40E2F55: do_pvhash (hash.c:371)
> ==6204==    by 0x40E2F55: pvhash_remove (hash.c:417)
> ==6204==    by 0x40739E6: registry_destroy_file (registry.c:255)
> ==6204==    by 0x4056C93: event_finalize (event.c:116)
> ==6204==    by 0x406A17A: eventobj_finalize (eventobj.c:150)
> ==6204==    by 0x406B022: __syncobj_finalize (syncobj.c:294)
> ==6204==    by 0x406BD75: syncobj_destroy (syncobj.c:573)
> ==6204==    by 0x406A290: eventobj_destroy (eventobj.c:182)
> ==6204==    by 0x4057017: rt_event_create (event.c:218)
> ==6204==    by 0x81942C0: RtosCreateNamedEvent (RtosXenomai3.c:279)
> ==6204==    by 0x80D67FD: CallbackInterruptEventOpen (Callbacks.c:488)

Ok, this one is a double-deletion bug in an error path from
rt_create_event(). Will fix, thanks.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/06/2015 05:00 PM, Charles Kiorpes wrote:
>> Could you --enable-debug, and have gdb determine the offending
>> instruction? TIA,
> 
> The gdb output is as follows:
> 
> Program received signal SIGSEGV, Segmentation fault.
> get_free_range (heap=heap@entry=0x6dc07000, bsize=bsize@entry=256, log2size=8)
>     at ../../../xenomai-3.0-rc6/lib/copperplate/heapobj-pshared.c:242
> 242                                           freepage =
> __shref_check(heap, *((memoff_t *)freepage));
> 
> Thanks,
> - Charles
> 

To debug this further, I would need the full backtrace of this bug using
gdb, and the value of the freepage variable. In fact, "info locals"
would help even better. Thanks,

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/10/2015 03:52 PM, Charles Kiorpes wrote:
>> This may be a memory corruption issue, some metadata from the shared
>> heap got trashed earlier in the init sequence.
>>
>> Would it be practical to switch this app to the mercury core (no need
>> for preempt-rt, just a plain regular desktop kernel), only for the
>> purpose of running the init phase over valgrind? That would entail
>> stubbing/bypassing the hardware-related inits (e.g. drivers) if ever
>> possible.
> 
> I reconfigured to use Mercury, but I can't reproduce the error in this
> configuration.  There is no longer any segfault, or ENOMEM.
> 
> Despite being unable to reproduce the problem, Valgrind output
> indicates something funny happening when I call rt_event_create() on
> an object that already exists.
> 
> Here's the output from Valgrind of what occurs when I call
> rt_event_create() in pshared mode with an event that already exists
> (I've truncated some of the call stacks, this is all from one call to
> rt_event_create through a wrapper layer):
> 

Fixed in the -next branch, thanks.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/10/2015 05:17 PM, Charles Kiorpes wrote:
>> Is this case (trying to create an event flag with a conflicting name)
>> part of the original scenario with Cobalt?
> 
> This rt_event_create() with existing name occurs a little earlier in
> my application's startup phase than where the segfault occurs using
> Cobalt.  However, applying the rt_event_bind() workaround and
> switching back to Cobalt does not change the segfault behavior (it
> still occurs at the same place), so I'm not convinced that they're
> related.
> 
> When I run the processes through Valgrind using the Mercury core and
> checking event existence with rt_event_bind() before attempting to
> create, Valgrind finds no issues.  The application hangs at a
> different location (creation of a periodic task; I'm fairly certain
> that I just need to rework that section of code a little).
> 
> However, my application really need the tight timings that Cobalt
> provides, and this has to eventually run on an embedded system, so
> switching to Mercury is not a viable solution.
> 

You should retry with the current head of the -next branch with Cobalt.
No guarantee, but several fixes took place in the shared heap
implementation. Since I don't know much about your application, I can
only assume that one of them might help. At the very least, this is a
better starting point than -rc6.

-- 
Philippe.

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Charles Kiorpes]

Hello Philippe,
I have spent the past few days stressing my application and trying to
get the problem to reproduce, so that I could better debug it and
provide more information.  However, my current configuration seems to
be fixed.

I made three changes (still using rc6):
1 - Fixed the small bug mentioned in your first patch in this thread
2 - Reordered event creation logic so that rt_event_bind() is used to
check existence before rt_event_create() is called
3 - Increased the heap sizes in the kernel configuration again
(system=1024, shared=256, private=256)

These changes have me up and running on an x86 system.

> You should retry with the current head of the -next branch with Cobalt.
I'll switch to the -next branch and continue testing.  I'm also doing
an ARM port of the app, and was having some small problems with shared
heap initialization, so maybe I can resume work there.

Thanks for all your help!

-Charles

Re: [Xenomai] Segfaults and ENOMEM during rt_event_create()

[Philippe Gerum]

On 08/13/2015 01:39 PM, Charles Kiorpes wrote:
> Hello Philippe,
> I have spent the past few days stressing my application and trying to
> get the problem to reproduce, so that I could better debug it and
> provide more information.  However, my current configuration seems to
> be fixed.
> 
> I made three changes (still using rc6):
> 1 - Fixed the small bug mentioned in your first patch in this thread
> 2 - Reordered event creation logic so that rt_event_bind() is used to
> check existence before rt_event_create() is called
> 3 - Increased the heap sizes in the kernel configuration again
> (system=1024, shared=256, private=256)
> 

I'd be interested to know whether the bug re-appears when this latter
increase is not done, only keeping #1 and #2.

> These changes have me up and running on an x86 system.
> 
>> You should retry with the current head of the -next branch with Cobalt.
> I'll switch to the -next branch and continue testing.  I'm also doing
> an ARM port of the app, and was having some small problems with shared
> heap initialization, so maybe I can resume work there.

It's great that you made some progress. Meanwhile, I reviewed the shared
heap management lately and some stuff there wasn't quite right. This is
the list of recent commits in the -next tree that may/might concern your
application, with some notes:

{rpm@raven} git log --reverse --oneline -13
23bb315 lib/cobalt: fix private event state accessor

This commit contains the patch mentioned in #1.

fab0540 kernel/cobalt: Add reason to gorelax trace point
6dfdc69 cobalt/arm: Detect software breakpoints

Unrelated stuff.

033976a lib/cobalt/heapobj: pshared: allow group access to heap

You need this one to bind several processes with different euids to a
single session, provided they all belong to a common gid.

fa3092a copperplate/semobj: add uninit cleanup helper
824d831 copperplate/eventobj: add uninit cleanup helper
9e3cbdb alchemy/event: prevent double-free on error path
ac6fb1c alchemy/sem: prevent double-free on error path

The four commits above fix the double deletion bug occurring upon an
attempt to create a sema4 or an event with a conflicting name, using the
Alchemy API. This may certainly have caused havoc in the heap when
happening. This should allow you to drop the work around based on
rt_event_bind() mentioned in #2.

f333317 vxworks/memPartLib: fix invalid status from addToPool
f928e4d vxworks/memPartLib: do not account for failed alloc

Some fixups for the Vxworks API users who depend on memory partitions.

65f16f4 copperplate/heapobj-pshared: fix and sanitize extent management

This one is addresses several issues in the shared heap management,
particularly when extending embedded heaps (like the VxWorks API does).
Alchemy did not use the extension service, but I would recommend to pull
those sanitizing changes nevertheless.

0da6b0d boilerplate/ancillaries: add routine to parse mem size
b151d3b copperplate/init: disambiguate --mem-pool-size argument

Not directly related. This said, this commit clarifies the argument to
--mem-pool-size. You probably want to pass --mem-pool-size=400k with
this patch in.

> 
> Thanks for all your help!
> 
> -Charles
> 


-- 
Philippe.