[Xenomai] Kernel Configuration Woes

[Xenomai] Kernel Configuration Woes

[Justin Hart]

I recently rebuilt my kernel and I seem to be running into some problems
that I previously encountered with xenomai.  Unforutnately, I can't seem to
fix them.

When I load my application (which runs at Barrett WAM), I get the following
messages in my kernel log.

Dec 19 23:59:01 magnum kernel: [ 1232.835449] Clocksource tsc unstable
(delta = 1998076024 ns)
Dec 19 23:59:01 magnum kernel: [ 1232.835477] Switched to clocksource
refined-jiffies

At this juncture, a few things happen, my entire system grinds to a halt,
my system clock starts to drift significantly, and xeno-test reports major
clock drift.

CPU  ToD offset [us] ToD drift [us/s]      warps max delta [us]
--- -------------------- ---------------- ---------- --------------
  0          164328553.0      7034205.095          0            0.0
  1          164329131.5      7033385.416          0            0.0
  2          164328794.0      7030756.296          0            0.0
  3          164328783.3      7030646.980          0            0.0

I'm running kernel 3.14.17, xenomai 2.6.4, and the default ipipe that comes
with 2.6.4

Attached is my .config

Does anybody have any idea what I'm doing wrong?  This looks pretty close
to what I had marked in my notes.

Justin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config
Type: application/octet-stream
Size: 93765 bytes
Desc: not available
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20141220/1c1c7c38/attachment.obj>

Re: [Xenomai] Kernel Configuration Woes

[Lennart Sorensen]

On Sat, Dec 20, 2014 at 12:05:25AM -0800, Justin Hart wrote:
> I recently rebuilt my kernel and I seem to be running into some problems
> that I previously encountered with xenomai.  Unforutnately, I can't seem to
> fix them.
> 
> When I load my application (which runs at Barrett WAM), I get the following
> messages in my kernel log.
> 
> Dec 19 23:59:01 magnum kernel: [ 1232.835449] Clocksource tsc unstable
> (delta = 1998076024 ns)
> Dec 19 23:59:01 magnum kernel: [ 1232.835477] Switched to clocksource
> refined-jiffies
> 
> At this juncture, a few things happen, my entire system grinds to a halt,
> my system clock starts to drift significantly, and xeno-test reports major
> clock drift.
> 
> CPU  ToD offset [us] ToD drift [us/s]      warps max delta [us]
> --- -------------------- ---------------- ---------- --------------
>   0          164328553.0      7034205.095          0            0.0
>   1          164329131.5      7033385.416          0            0.0
>   2          164328794.0      7030756.296          0            0.0
>   3          164328783.3      7030646.980          0            0.0
> 
> I'm running kernel 3.14.17, xenomai 2.6.4, and the default ipipe that comes
> with 2.6.4
> 
> Attached is my .config
> 
> Does anybody have any idea what I'm doing wrong?  This looks pretty close
> to what I had marked in my notes.

I suspect it would help a lot to know what hardware this is on.

The tsc being unstable seems rather bad, since I can't imagine using
jiffies is going to give any removely stable for system time.

I am surprised if any modern x86 hardware has bad tsc support, unless
of course SMI is involved.

I suspect if you figure out why the tsc is being declared unstable, you
will fix the problem.

-- 
Len Sorensen

Re: [Xenomai] Kernel Configuration Woes

[Gilles Chanteperdrix]

On Sat, Dec 20, 2014 at 10:33:37AM -0500, Lennart Sorensen wrote:
> On Sat, Dec 20, 2014 at 12:05:25AM -0800, Justin Hart wrote:
> > I recently rebuilt my kernel and I seem to be running into some problems
> > that I previously encountered with xenomai.  Unforutnately, I can't seem to
> > fix them.
> > 
> > When I load my application (which runs at Barrett WAM), I get the following
> > messages in my kernel log.
> > 
> > Dec 19 23:59:01 magnum kernel: [ 1232.835449] Clocksource tsc unstable
> > (delta = 1998076024 ns)
> > Dec 19 23:59:01 magnum kernel: [ 1232.835477] Switched to clocksource
> > refined-jiffies
> > 
> > At this juncture, a few things happen, my entire system grinds to a halt,
> > my system clock starts to drift significantly, and xeno-test reports major
> > clock drift.
> > 
> > CPU  ToD offset [us] ToD drift [us/s]      warps max delta [us]
> > --- -------------------- ---------------- ---------- --------------
> >   0          164328553.0      7034205.095          0            0.0
> >   1          164329131.5      7033385.416          0            0.0
> >   2          164328794.0      7030756.296          0            0.0
> >   3          164328783.3      7030646.980          0            0.0
> > 
> > I'm running kernel 3.14.17, xenomai 2.6.4, and the default ipipe that comes
> > with 2.6.4
> > 
> > Attached is my .config
> > 
> > Does anybody have any idea what I'm doing wrong?  This looks pretty close
> > to what I had marked in my notes.
> 
> I suspect it would help a lot to know what hardware this is on.
> 
> The tsc being unstable seems rather bad, since I can't imagine using
> jiffies is going to give any removely stable for system time.
> 
> I am surprised if any modern x86 hardware has bad tsc support, unless
> of course SMI is involved.

Very likely the problem is not related to the tsc at all. You have
to understand that the principle of Xenomai is to stop Linux at some
point to run the RT threads, then resume Linux at the point where it
was preempted when the RT threads suspend. So, if the RT threads run
for too long, Linux will experience some "loss of time", which may
have some consequences on all sorts of watchdogs, the tsc having
such a watchdog. The usual advice is to get Xenomai threads to
suspend every Linux tick. I believe the tsc watchdog has a
workaround for this issue in the I-pipe patch, but maybe it got
broken by some upstream code change in 3.14.

-- 
					    Gilles.

Re: [Xenomai] Kernel Configuration Woes

[Gilles Chanteperdrix]

On Sat, Dec 20, 2014 at 10:33:37AM -0500, Lennart Sorensen wrote:
> On Sat, Dec 20, 2014 at 12:05:25AM -0800, Justin Hart wrote:
> > I recently rebuilt my kernel and I seem to be running into some problems
> > that I previously encountered with xenomai.  Unforutnately, I can't seem to
> > fix them.
> > 
> > When I load my application (which runs at Barrett WAM), I get the following
> > messages in my kernel log.
> > 
> > Dec 19 23:59:01 magnum kernel: [ 1232.835449] Clocksource tsc unstable
> > (delta = 1998076024 ns)
> > Dec 19 23:59:01 magnum kernel: [ 1232.835477] Switched to clocksource
> > refined-jiffies
> > 
> > At this juncture, a few things happen, my entire system grinds to a halt,
> > my system clock starts to drift significantly, and xeno-test reports major
> > clock drift.
> > 
> > CPU  ToD offset [us] ToD drift [us/s]      warps max delta [us]
> > --- -------------------- ---------------- ---------- --------------
> >   0          164328553.0      7034205.095          0            0.0
> >   1          164329131.5      7033385.416          0            0.0
> >   2          164328794.0      7030756.296          0            0.0
> >   3          164328783.3      7030646.980          0            0.0
> > 
> > I'm running kernel 3.14.17, xenomai 2.6.4, and the default ipipe that comes
> > with 2.6.4
> > 
> > Attached is my .config
> > 
> > Does anybody have any idea what I'm doing wrong?  This looks pretty close
> > to what I had marked in my notes.
> 
> I suspect it would help a lot to know what hardware this is on.
> 
> The tsc being unstable seems rather bad, since I can't imagine using
> jiffies is going to give any removely stable for system time.
> 
> I am surprised if any modern x86 hardware has bad tsc support, unless
> of course SMI is involved.

To make it simple: from Linux point of view, Xenomai behaves just as
an SMI.

-- 
					    Gilles.

Re: [Xenomai] Kernel Configuration Woes

[Lennart Sorensen]

On Sat, Dec 20, 2014 at 05:04:08PM +0100, Gilles Chanteperdrix wrote:
> To make it simple: from Linux point of view, Xenomai behaves just as
> an SMI.

Certainly true, but that's the software you control so you can make it
behave itself.

-- 
Len Sorensen

Re: [Xenomai] Kernel Configuration Woes

[Gilles Chanteperdrix]

On Sat, Dec 20, 2014 at 11:21:39AM -0500, Lennart Sorensen wrote:
> On Sat, Dec 20, 2014 at 05:04:08PM +0100, Gilles Chanteperdrix wrote:
> > To make it simple: from Linux point of view, Xenomai behaves just as
> > an SMI.
> 
> Certainly true, but that's the software you control so you can make it
> behave itself.

Well, the only way to make it behave itself is to not run it at all,
because as soon as Xenomai run its threads, it "steals" time to
Linux. So, the workaround is to get it to run not too long, but it
is not guaranteed to be sufficient to avoid detection, and the other
workaround is to disable Linux detection mechanisms.

-- 
					    Gilles.

Re: [Xenomai] Kernel Configuration Woes

[Justin Hart]

Sorry for the delayed reply!  I was a bit under the weather this weekend.
I think that you folks have hit on my alternate hypothesis: that I am doing
something in my program in the real-time thread that is causing problems.
It's a little weird, because I am linking against libbarrett.  I wrote a
wrapper to the arm code that gives me a simple simulated arm or the real
arm, based on how the program is invoked.  When it's not the real arm,
however, nothing that interacts with xenomai is launched.  I'm going to do
some testing today and see if I can narrow in on the problem.  I might have
some more questions.  Thanks for the help so far, though!

On Sat, Dec 20, 2014 at 8:29 AM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> On Sat, Dec 20, 2014 at 11:21:39AM -0500, Lennart Sorensen wrote:
> > On Sat, Dec 20, 2014 at 05:04:08PM +0100, Gilles Chanteperdrix wrote:
> > > To make it simple: from Linux point of view, Xenomai behaves just as
> > > an SMI.
> >
> > Certainly true, but that's the software you control so you can make it
> > behave itself.
>
> Well, the only way to make it behave itself is to not run it at all,
> because as soon as Xenomai run its threads, it "steals" time to
> Linux. So, the workaround is to get it to run not too long, but it
> is not guaranteed to be sufficient to avoid detection, and the other
> workaround is to disable Linux detection mechanisms.
>
> --
>                                             Gilles.
>

Re: [Xenomai] Kernel Configuration Woes

[Justin Hart]

Oh, Lennart.  Commodity hardware.  Core i7, 32G RAM, Ubuntu 14.04, kernel
as described above, Linux 3.14.17 w/ Xenomai 2.6.4 with the packaged Adeos
patch.  The only thing interesting about the configuration that I thought
might be contributing to issues is running the CUDA driver.  I stumbled
onto the hypothesis that something was wrong with the kernel because I was
having rendering issues.  Backing out CUDA appeared to resolve the issues,
but I think that it's my software doing something *really* boneheaded.  I
have a small GUI that I wrote that displays the current pose of the arm.  I
think that I'm refreshing a set of gtk_entries with gtk_entry_set_text
every time the arm's position is updated.  I *should* be doing this in a
GTK worker that just updates it every time it goes through gtk_main_loop.
I think that what I'm doing is simply giving too many events to the
x-server.  I first noticed this because there was some latency in the arm's
responsiveness.

Altering it to test this should be easy enough, though.

Justin

On Mon, Dec 22, 2014 at 11:57 AM, Justin Hart <justin.hart@mech.ubc.ca>
wrote:

> Sorry for the delayed reply!  I was a bit under the weather this weekend.
> I think that you folks have hit on my alternate hypothesis: that I am doing
> something in my program in the real-time thread that is causing problems.
> It's a little weird, because I am linking against libbarrett.  I wrote a
> wrapper to the arm code that gives me a simple simulated arm or the real
> arm, based on how the program is invoked.  When it's not the real arm,
> however, nothing that interacts with xenomai is launched.  I'm going to do
> some testing today and see if I can narrow in on the problem.  I might have
> some more questions.  Thanks for the help so far, though!
>
> On Sat, Dec 20, 2014 at 8:29 AM, Gilles Chanteperdrix <
> gilles.chanteperdrix@xenomai.org> wrote:
>
>> On Sat, Dec 20, 2014 at 11:21:39AM -0500, Lennart Sorensen wrote:
>> > On Sat, Dec 20, 2014 at 05:04:08PM +0100, Gilles Chanteperdrix wrote:
>> > > To make it simple: from Linux point of view, Xenomai behaves just as
>> > > an SMI.
>> >
>> > Certainly true, but that's the software you control so you can make it
>> > behave itself.
>>
>> Well, the only way to make it behave itself is to not run it at all,
>> because as soon as Xenomai run its threads, it "steals" time to
>> Linux. So, the workaround is to get it to run not too long, but it
>> is not guaranteed to be sufficient to avoid detection, and the other
>> workaround is to disable Linux detection mechanisms.
>>
>> --
>>                                             Gilles.
>>
>
>

Re: [Xenomai] Kernel Configuration Woes

[Lennart Sorensen]

On Mon, Dec 22, 2014 at 12:05:35PM -0800, Justin Hart wrote:
> Oh, Lennart.  Commodity hardware.  Core i7, 32G RAM, Ubuntu 14.04, kernel
> as described above, Linux 3.14.17 w/ Xenomai 2.6.4 with the packaged Adeos
> patch.  The only thing interesting about the configuration that I thought
> might be contributing to issues is running the CUDA driver.  I stumbled
> onto the hypothesis that something was wrong with the kernel because I was
> having rendering issues.  Backing out CUDA appeared to resolve the issues,
> but I think that it's my software doing something *really* boneheaded.  I
> have a small GUI that I wrote that displays the current pose of the arm.  I
> think that I'm refreshing a set of gtk_entries with gtk_entry_set_text
> every time the arm's position is updated.  I *should* be doing this in a
> GTK worker that just updates it every time it goes through gtk_main_loop.
> I think that what I'm doing is simply giving too many events to the
> x-server.  I first noticed this because there was some latency in the arm's
> responsiveness.
> 
> Altering it to test this should be easy enough, though.

Well as was mentioned recently on the list, if you use the nvidia kernel
modules, then you really can not expect xenomai/ipipe to work properly,
because you can't fix what the binary blob from nvidia does, and it
could easily mess up interrupt handling and make the realtime stop
being realtime.

-- 
Len Sorensen

Re: [Xenomai] Kernel Configuration Woes

[Justin Hart]

Yeah, I saw that.  I may need to back that out and put in the open source
driver.

On Mon, Dec 22, 2014 at 12:22 PM, Lennart Sorensen <
lsorense@csclub.uwaterloo.ca> wrote:

> On Mon, Dec 22, 2014 at 12:05:35PM -0800, Justin Hart wrote:
> > Oh, Lennart.  Commodity hardware.  Core i7, 32G RAM, Ubuntu 14.04, kernel
> > as described above, Linux 3.14.17 w/ Xenomai 2.6.4 with the packaged
> Adeos
> > patch.  The only thing interesting about the configuration that I thought
> > might be contributing to issues is running the CUDA driver.  I stumbled
> > onto the hypothesis that something was wrong with the kernel because I
> was
> > having rendering issues.  Backing out CUDA appeared to resolve the
> issues,
> > but I think that it's my software doing something *really* boneheaded.  I
> > have a small GUI that I wrote that displays the current pose of the
> arm.  I
> > think that I'm refreshing a set of gtk_entries with gtk_entry_set_text
> > every time the arm's position is updated.  I *should* be doing this in a
> > GTK worker that just updates it every time it goes through gtk_main_loop.
> > I think that what I'm doing is simply giving too many events to the
> > x-server.  I first noticed this because there was some latency in the
> arm's
> > responsiveness.
> >
> > Altering it to test this should be easy enough, though.
>
> Well as was mentioned recently on the list, if you use the nvidia kernel
> modules, then you really can not expect xenomai/ipipe to work properly,
> because you can't fix what the binary blob from nvidia does, and it
> could easily mess up interrupt handling and make the realtime stop
> being realtime.
>
> --
> Len Sorensen
>

Re: [Xenomai] Kernel Configuration Woes

[Justin Hart]

I figured it out.  One of my threads was not cleanly exiting and was
screwing up the OS.  I more carefully cleaned up at exit and life is good.

Thanks folks!

On Mon, Dec 22, 2014 at 1:04 PM, Justin Hart <justin.hart@mech.ubc.ca>
wrote:

> Yeah, I saw that.  I may need to back that out and put in the open source
> driver.
>
> On Mon, Dec 22, 2014 at 12:22 PM, Lennart Sorensen <
> lsorense@csclub.uwaterloo.ca> wrote:
>
>> On Mon, Dec 22, 2014 at 12:05:35PM -0800, Justin Hart wrote:
>> > Oh, Lennart.  Commodity hardware.  Core i7, 32G RAM, Ubuntu 14.04,
>> kernel
>> > as described above, Linux 3.14.17 w/ Xenomai 2.6.4 with the packaged
>> Adeos
>> > patch.  The only thing interesting about the configuration that I
>> thought
>> > might be contributing to issues is running the CUDA driver.  I stumbled
>> > onto the hypothesis that something was wrong with the kernel because I
>> was
>> > having rendering issues.  Backing out CUDA appeared to resolve the
>> issues,
>> > but I think that it's my software doing something *really* boneheaded.
>> I
>> > have a small GUI that I wrote that displays the current pose of the
>> arm.  I
>> > think that I'm refreshing a set of gtk_entries with gtk_entry_set_text
>> > every time the arm's position is updated.  I *should* be doing this in a
>> > GTK worker that just updates it every time it goes through
>> gtk_main_loop.
>> > I think that what I'm doing is simply giving too many events to the
>> > x-server.  I first noticed this because there was some latency in the
>> arm's
>> > responsiveness.
>> >
>> > Altering it to test this should be easy enough, though.
>>
>> Well as was mentioned recently on the list, if you use the nvidia kernel
>> modules, then you really can not expect xenomai/ipipe to work properly,
>> because you can't fix what the binary blob from nvidia does, and it
>> could easily mess up interrupt handling and make the realtime stop
>> being realtime.
>>
>> --
>> Len Sorensen
>>
>
>