[Xenomai] slackspot

[Xenomai] slackspot

[Lowell Gilbert]

I have been unable to use slackspot, because /proc/xenomai/debug/relax
is empty. I do have CONFIG_XENO_OPT_DEBUG_TRACE_RELAX enabled (without
that enabled, I don't even have the file). I have not run /bin/relax,
mentioned in the manual page, because I don't have such a program and
can't find any reference to it.

I am using 3.0.2.

Thanks.

Re: [Xenomai] slackspot

[Philippe Gerum]

On 06/16/2016 04:43 PM, Lowell Gilbert wrote:
> I have been unable to use slackspot, because /proc/xenomai/debug/relax
> is empty. I do have CONFIG_XENO_OPT_DEBUG_TRACE_RELAX enabled (without
> that enabled, I don't even have the file).

This is expected.

 I have not run /bin/relax,
> mentioned in the manual page, because I don't have such a program and
> can't find any reference to it.
> 
> I am using 3.0.2.


/bin/relax is only the name of a fake executable that would have
produced the typical output described in the example, read /foo/bar if
that helps.

Once TRACE_RELAX is enabled in the kernel, running any application that
causes switches to secondary mode should populate
/proc/xenomai/debug/relax with event records.

-- 
Philippe.

Re: [Xenomai] slackspot

[Lowell Gilbert]

Philippe Gerum <rpm@xenomai.org> writes:

> On 06/16/2016 04:43 PM, Lowell Gilbert wrote:
>>  I have not run /bin/relax,
>> mentioned in the manual page, because I don't have such a program and
>> can't find any reference to it.
>> 
>> I am using 3.0.2.
>
>
> /bin/relax is only the name of a fake executable that would have
> produced the typical output described in the example, read /foo/bar if
> that helps.

That seems obvious now that you've explained it. 

Maybe something like the following would help:
--- doc/asciidoc/man1/slackspot.adoc.~1~ 2016-02-18 07:17:40.000000000
-0500
+++ doc/asciidoc/man1/slackspot.adoc    2016-06-20 11:16:22.130252140
-0400
@@ -91,7 +91,7 @@

 In the following scenario, the _target_ system built with the
  CONFIG_XENO_OPT_DEBUG_TRACE_RELAX feature enabled in the kernel
  -configuration, just ran the _/bin/relax_ program.
  +configuration, just ran the Xenomai-enabled _/bin/relax_ program.

 This program caused a transition to secondary mode switch of the
  current task (_Task 2_) as a result of calling +putchar()+. The Cobalt

> Once TRACE_RELAX is enabled in the kernel, running any application that
> causes switches to secondary mode should populate
> /proc/xenomai/debug/relax with event records.

Hmm. I'm still not getting the records, and the SIGXCPU method isn't
giving me anything either. But the MSW, CSW, and XSC counts are all
rising at exactly the rate that the thread wakes up.

Here's the design: I have a POSIX thread which calls into an RTDM ioctl
which blocks on an RTDM event, then copies a couple of dozen bytes of
data back to the POSIX thread. The POSIX thread does some calculations
on the data, then uses another ioctl to write the results back to the
hardware. And repeats.

Is it possible that the thread might be relaxing in the kernel (RTDM
ioctl) and not receiving signals as as result?

I have two theories about what is happening.
   1. I have made an error somewhere that is resulting in unnecessary
      relax events. In that case, getting the stack traces is the key to
      finding said error. I can create a small example if that helps me
      get assistance in looking at it.
   2. Having a real-time pthread block in an RTDM ioctl may be an
      inherently bad idea. I notice that all synchronization primitives
      are characterized by "might-switch". Perhaps I should be using a
      device read to block the POSIX thread instead of doing it in an
      ioctl()?

My current plan is to use the I-pipe tracer, but if there is an easier
way to move forward, I would appreciate advice to that effect.

Thank you very much.

 - Lowell Gilbert

Re: [Xenomai] slackspot

[Gilles Chanteperdrix]

On Mon, Jun 20, 2016 at 11:19:49AM -0400, Lowell Gilbert wrote:
> Philippe Gerum <rpm@xenomai.org> writes:
> 
> > On 06/16/2016 04:43 PM, Lowell Gilbert wrote:
> >>  I have not run /bin/relax,
> >> mentioned in the manual page, because I don't have such a program and
> >> can't find any reference to it.
> >> 
> >> I am using 3.0.2.
> >
> >
> > /bin/relax is only the name of a fake executable that would have
> > produced the typical output described in the example, read /foo/bar if
> > that helps.
> 
> That seems obvious now that you've explained it. 
> 
> Maybe something like the following would help:
> --- doc/asciidoc/man1/slackspot.adoc.~1~ 2016-02-18 07:17:40.000000000
> -0500
> +++ doc/asciidoc/man1/slackspot.adoc    2016-06-20 11:16:22.130252140
> -0400
> @@ -91,7 +91,7 @@
> 
>  In the following scenario, the _target_ system built with the
>   CONFIG_XENO_OPT_DEBUG_TRACE_RELAX feature enabled in the kernel
>   -configuration, just ran the _/bin/relax_ program.
>   +configuration, just ran the Xenomai-enabled _/bin/relax_ program.
> 
>  This program caused a transition to secondary mode switch of the
>   current task (_Task 2_) as a result of calling +putchar()+. The Cobalt
> 
> > Once TRACE_RELAX is enabled in the kernel, running any application that
> > causes switches to secondary mode should populate
> > /proc/xenomai/debug/relax with event records.
> 
> Hmm. I'm still not getting the records, and the SIGXCPU method isn't
> giving me anything either. But the MSW, CSW, and XSC counts are all
> rising at exactly the rate that the thread wakes up.
> 
> Here's the design: I have a POSIX thread which calls into an RTDM ioctl
> which blocks on an RTDM event, then copies a couple of dozen bytes of
> data back to the POSIX thread. The POSIX thread does some calculations
> on the data, then uses another ioctl to write the results back to the
> hardware. And repeats.
> 
> Is it possible that the thread might be relaxing in the kernel (RTDM
> ioctl) and not receiving signals as as result?
> 
> I have two theories about what is happening.
>    1. I have made an error somewhere that is resulting in unnecessary
>       relax events. In that case, getting the stack traces is the key to
>       finding said error. I can create a small example if that helps me
>       get assistance in looking at it.
>    2. Having a real-time pthread block in an RTDM ioctl may be an
>       inherently bad idea. I notice that all synchronization primitives
>       are characterized by "might-switch". Perhaps I should be using a
>       device read to block the POSIX thread instead of doing it in an
>       ioctl()?
> 
> My current plan is to use the I-pipe tracer, but if there is an easier
> way to move forward, I would appreciate advice to that effect.

There is one possibility: if your thread runs with the
SCHED_OTHER/SCHED_WEAK scheduling policy, I would not expect the
relaxes to trigger anything, as you have been asking for them by
choosing to use these policies.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] slackspot

[Lowell Gilbert]

Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

> There is one possibility: if your thread runs with the
> SCHED_OTHER/SCHED_WEAK scheduling policy, I would not expect the
> relaxes to trigger anything, as you have been asking for them by
> choosing to use these policies.

Yes, I should have mentioned that topic. I have run this under both
SCHED_FIFO and SCHED_RR.

Re: [Xenomai] slackspot

[Gilles Chanteperdrix]

On Mon, Jun 20, 2016 at 01:15:38PM -0400, Lowell Gilbert wrote:
> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
> 
> > There is one possibility: if your thread runs with the
> > SCHED_OTHER/SCHED_WEAK scheduling policy, I would not expect the
> > relaxes to trigger anything, as you have been asking for them by
> > choosing to use these policies.
> 
> Yes, I should have mentioned that topic. I have run this under both
> SCHED_FIFO and SCHED_RR.

Ok. Could you post an example allowing us to reproduce the issue ?


-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] slackspot

[Lowell Gilbert]

Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

>
> Ok. Could you post an example allowing us to reproduce the issue ?

I would be happy to do so.

Re: [Xenomai] slackspot

[Lowell Gilbert]

Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

> On Mon, Jun 20, 2016 at 01:15:38PM -0400, Lowell Gilbert wrote:
>> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
>> 
>> > There is one possibility: if your thread runs with the
>> > SCHED_OTHER/SCHED_WEAK scheduling policy, I would not expect the
>> > relaxes to trigger anything, as you have been asking for them by
>> > choosing to use these policies.
>> 
>> Yes, I should have mentioned that topic. I have run this under both
>> SCHED_FIFO and SCHED_RR.
>
> Ok. Could you post an example allowing us to reproduce the issue ?

I can. It took me a little longer than I figured because I had
accidentally deleted the MODULE_LICENSE. Is it possible to get the
kernel makefiles to tell you *why* it isn't linking in libraries you
asked for?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: example.tar
Type: archive/tar
Size: 10240 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20160621/eb21f260/attachment.bin>
-------------- next part --------------
I'm attaching a tar file, but in case this list strips attachments, it's
also at
 http://be-well.ilk.org/~lowell/example.tar
It's about a hundred lines each for the application and kernel module.

Thank you.

Re: [Xenomai] slackspot

[Lowell Gilbert]

Lowell Gilbert <kludge@be-well.ilk.org> writes:

> I'm attaching a tar file, but in case this list strips attachments, it's
> also at
>  http://be-well.ilk.org/~lowell/example.tar
> It's about a hundred lines each for the application and kernel module.

I have updated the tarball with a missing header file.

This time, I *tested* the tarball.
I apologize for the oversight.

Be well.

Re: [Xenomai] slackspot

[Philippe Gerum]

On 06/21/2016 08:59 PM, Lowell Gilbert wrote:
> Lowell Gilbert <kludge@be-well.ilk.org> writes:
> 
>> I'm attaching a tar file, but in case this list strips attachments, it's
>> also at
>>  http://be-well.ilk.org/~lowell/example.tar
>> It's about a hundred lines each for the application and kernel module.
> 
> I have updated the tarball with a missing header file.
> 
> This time, I *tested* the tarball.
> I apologize for the oversight.
> 

No problem, we are doing this all the time. Actually Xenomai is most
likely a massive collection of oversights that tend to be working well
together.

You app is missing a call to pthread_setmode_np() for enabling the
warn-upon-switch bit,
e.g. calling pthread_set_mode_np(0, PTHREAD_WARNSW, NULL) would enable
tracing for the calling thread.

http://www.xenomai.org/documentation/xenomai-3/html/xeno3prm/group__cobalt__api__thread.html#gae3b7df7f77c04253ed19fb6346f0f9b2

-- 
Philippe.

Re: [Xenomai] slackspot

[Lowell Gilbert]

Philippe Gerum <rpm@xenomai.org> writes:

> You app is missing a call to pthread_setmode_np() for enabling the
> warn-upon-switch bit,
> e.g. calling pthread_set_mode_np(0, PTHREAD_WARNSW, NULL) would enable
> tracing for the calling thread.

My real application has that. My example stripped out everything I could
live without, including lots of error-checking for errors that generally
don't come up.

I did actually test setting PTHREAD_WARNSW in the example program, and
there was no data to be read from /proc/xenomai/debug/relax.

Be well.

Re: [Xenomai] slackspot

[Lowell Gilbert]

Lowell Gilbert <kludge@be-well.ilk.org> writes:

> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
>> Ok. Could you post an example allowing us to reproduce the issue ?
>
> I can. It took me a little longer than I figured because I had
> accidentally deleted the MODULE_LICENSE. Is it possible to get the
> kernel makefiles to tell you *why* it isn't linking in libraries you
> asked for?

>  http://be-well.ilk.org/~lowell/example.tar
> It's about a hundred lines each for the application and kernel module.

Has anyone tried this out? My real application is meeting its latency
requirements, but I'm pretty sure it's having more of an impact on the
rest of the system than it needs to do.

Be well.

Re: [Xenomai] slackspot

[Lowell Gilbert]

Lowell Gilbert <kludge@be-well.ilk.org> writes:

> Lowell Gilbert <kludge@be-well.ilk.org> writes:
>
>> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
>>> Ok. Could you post an example allowing us to reproduce the issue ?
>>
>> I can. It took me a little longer than I figured because I had
>> accidentally deleted the MODULE_LICENSE. Is it possible to get the
>> kernel makefiles to tell you *why* it isn't linking in libraries you
>> asked for?
>
>>  http://be-well.ilk.org/~lowell/example.tar
>> It's about a hundred lines each for the application and kernel module.
>
> Has anyone tried this out? My real application is meeting its latency
> requirements, but I'm pretty sure it's having more of an impact on the
> rest of the system than it needs to do.

I've tried changing the event to a semaphore, and changing the "read"
ioctl to a device read, and I'm still seeing huge numbers of state switches.

Re: [Xenomai] slackspot

[George Broz]

> Lowell Gilbert <kludge@be-well.ilk.org> writes:
>
>> Lowell Gilbert <kludge@be-well.ilk.org> writes:
>>
>>> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
>>>> Ok. Could you post an example allowing us to reproduce the issue ?
>>>
>>> I can. It took me a little longer than I figured because I had
>>> accidentally deleted the MODULE_LICENSE. Is it possible to get the
>>> kernel makefiles to tell you *why* it isn't linking in libraries you
>>> asked for?
>>
>>>  http://be-well.ilk.org/~lowell/example.tar
>>> It's about a hundred lines each for the application and kernel module.
>>
>> Has anyone tried this out? My real application is meeting its latency
>> requirements, but I'm pretty sure it's having more of an impact on the
>> rest of the system than it needs to do.
>
> I've tried changing the event to a semaphore, and changing the "read"
> ioctl to a device read, and I'm still seeing huge numbers of state switches.

Hi Lowell,

Did you ever figure out how to get slackspot working? I'm experiencing the same
issue - debug/relax is empty but the MSW counter is regularly incrementing
for my Xenomai application. Using 3.0.2 on ARM. Enabled XENO_.._TRACE_RELAX
in the kernel and set PTHREAD_WARNSW on the task. Any help appreciated.


Thanks,
--George

Re: [Xenomai] slackspot

[Alex Plits]

Hi,

Had the same issue (ARMv7/Cobalt/3.0.2), did not find any solution, fail back to debugging using the SIGDEBUG signaling catching.

Best Regards,
Alex Plits.

-----Original Message-----
From: Xenomai [mailto:xenomai-bounces@xenomai.org] On Behalf Of George Broz
Sent: יום ג 23 אוגוסט 2016 04:35
To: Lowell Gilbert <kludge@be-well.ilk.org>
Cc: xenomai@xenomai.org
Subject: Re: [Xenomai] slackspot

> Lowell Gilbert <kludge@be-well.ilk.org> writes:
>
>> Lowell Gilbert <kludge@be-well.ilk.org> writes:
>>
>>> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
>>>> Ok. Could you post an example allowing us to reproduce the issue ?
>>>
>>> I can. It took me a little longer than I figured because I had 
>>> accidentally deleted the MODULE_LICENSE. Is it possible to get the 
>>> kernel makefiles to tell you *why* it isn't linking in libraries you 
>>> asked for?
>>
>>>  http://be-well.ilk.org/~lowell/example.tar
>>> It's about a hundred lines each for the application and kernel module.
>>
>> Has anyone tried this out? My real application is meeting its latency 
>> requirements, but I'm pretty sure it's having more of an impact on 
>> the rest of the system than it needs to do.
>
> I've tried changing the event to a semaphore, and changing the "read"
> ioctl to a device read, and I'm still seeing huge numbers of state switches.

Hi Lowell,

Did you ever figure out how to get slackspot working? I'm experiencing the same issue - debug/relax is empty but the MSW counter is regularly incrementing for my Xenomai application. Using 3.0.2 on ARM. Enabled XENO_.._TRACE_RELAX in the kernel and set PTHREAD_WARNSW on the task. Any help appreciated.


Thanks,
--George

_______________________________________________
Xenomai mailing list
Xenomai@xenomai.org
https://xenomai.org/mailman/listinfo/xenomai

Re: [Xenomai] slackspot

[George Broz]

On 23 August 2016 at 10:14, Alex Plits <alex_plits@radwin.com> wrote:
> Hi,
>
> Had the same issue (ARMv7/Cobalt/3.0.2), did not find any solution, fail back to debugging using the SIGDEBUG signaling catching.
>
> Best Regards,
> Alex Plits.
>
> -----Original Message-----

Thanks.

Has anyone out there successfully run slackspot on ARM? After a day of
trying with a very simple single task example, I'm giving up.

Regarding SIGDEBUG - I started with the code from here:

https://xenomai.org/2014/06/finding-spurious-relaxes/

but found I had to change signal(sig, SIG_DFL); to signal(sig,
SIG_IGN); to prevent the application from exiting on the first
instance
of a (non-lethal) secondary transition (and using sigdebug_reason(si)
to return the correct code). With SIG_IGN, the application does
not terminate, but the SIGDEBUG handler is not invoked for subsequent
secondary transition events (as indicated by the MSW counter).

Is there a trick to catching multiple SIGDEBUG events on the same task?


Thanks in advance,
--George

Re: [Xenomai] slackspot

[Philippe Gerum]

On 08/24/2016 02:35 AM, George Broz wrote:
> On 23 August 2016 at 10:14, Alex Plits <alex_plits@radwin.com> wrote:
>> Hi,
>>
>> Had the same issue (ARMv7/Cobalt/3.0.2), did not find any solution, fail back to debugging using the SIGDEBUG signaling catching.
>>
>> Best Regards,
>> Alex Plits.
>>
>> -----Original Message-----
> 
> Thanks.
> 
> Has anyone out there successfully run slackspot on ARM? After a day of
> trying with a very simple single task example, I'm giving up.
> 
> Regarding SIGDEBUG - I started with the code from here:
> 
> https://xenomai.org/2014/06/finding-spurious-relaxes/
> 
> but found I had to change signal(sig, SIG_DFL); to signal(sig,
> SIG_IGN); to prevent the application from exiting on the first
> instance
> of a (non-lethal) secondary transition (and using sigdebug_reason(si)
> to return the correct code). With SIG_IGN, the application does
> not terminate, but the SIGDEBUG handler is not invoked for subsequent
> secondary transition events (as indicated by the MSW counter).
> 
> Is there a trick to catching multiple SIGDEBUG events on the same task?
> 
> 

Yes, just don't change the re-trap the signal in the handler, leave it
as is. testsuite/latency.c illustrates this, we trap SIGDEBUG to SIG_DFL
only when the -b option was given (stop_upon_switch), to make sure the
default action will be taken when we resend it for exiting.

-- 
Philippe.

Re: [Xenomai] slackspot

[Philippe Gerum]

On 08/24/2016 12:14 PM, Philippe Gerum wrote:
> On 08/24/2016 02:35 AM, George Broz wrote:
>> On 23 August 2016 at 10:14, Alex Plits <alex_plits@radwin.com> wrote:
>>> Hi,
>>>
>>> Had the same issue (ARMv7/Cobalt/3.0.2), did not find any solution, fail back to debugging using the SIGDEBUG signaling catching.
>>>
>>> Best Regards,
>>> Alex Plits.
>>>
>>> -----Original Message-----
>>
>> Thanks.
>>
>> Has anyone out there successfully run slackspot on ARM? After a day of
>> trying with a very simple single task example, I'm giving up.
>>
>> Regarding SIGDEBUG - I started with the code from here:
>>
>> https://xenomai.org/2014/06/finding-spurious-relaxes/
>>
>> but found I had to change signal(sig, SIG_DFL); to signal(sig,
>> SIG_IGN); to prevent the application from exiting on the first
>> instance
>> of a (non-lethal) secondary transition (and using sigdebug_reason(si)
>> to return the correct code). With SIG_IGN, the application does
>> not terminate, but the SIGDEBUG handler is not invoked for subsequent
>> secondary transition events (as indicated by the MSW counter).
>>
>> Is there a trick to catching multiple SIGDEBUG events on the same task?
>>
>>
> 
> Yes, just don't change the re-trap the signal in the handler, leave it

Wow, ok, please read: just don't re-trap the signal in the handler.

-- 
Philippe.