rt-preempt: problem compiling rt-preempt 2.6.23.1-rt11 on MIPS

rt-preempt: problem compiling rt-preempt 2.6.23.1-rt11 on MIPS

[Tim Bird]

I applied the patches in patch-2.6.23.1-rt11-broken-out.tar.bz2
to a Linux kernel version 2.6.23.1 (along with a few other
board specific patches).

I got the following compilation error:

  GEN     /home/tbird/work/rt-preempt/build/tx49/Makefile
  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CALL    /home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/scripts/checksyscalls.sh
  CHK     include/linux/compile.h
  CC      kernel/latency_trace.o
/home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/kernel/latency_trace.c:28:21: error: asm/rtc.h: No such file or directory
make[2]: *** [kernel/latency_trace.o] Error 1
make[1]: *** [kernel] Error 2
make: *** [vmlinux] Error 2

Indeed, there is no include/asm-mips/rtc.h.

I commented out the include line in latency_trace.c, and everything
compiled fine.  I'm not sure what is needed in an arch-specific rtc.h,
but compiling without it for the mips arch caused no problems.
Should I create a patch with a stub for rtc.h for mips?

As an aside, this has me worried.  Is anyone else doing any
RT Preempt testing or work on MIPS platforms, or am I forging
new ground? :-)

 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: rt-preempt: problem compiling rt-preempt 2.6.23.1-rt11 on MIPS

[Thomas Gleixner]

On Wed, 14 Nov 2007, Tim Bird wrote:

Added Steven and John to CC

> I applied the patches in patch-2.6.23.1-rt11-broken-out.tar.bz2
> to a Linux kernel version 2.6.23.1 (along with a few other
> board specific patches).
> 
> I got the following compilation error:
> 
>   GEN     /home/tbird/work/rt-preempt/build/tx49/Makefile
>   CHK     include/linux/version.h
>   CHK     include/linux/utsrelease.h
>   CALL    /home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/scripts/checksyscalls.sh
>   CHK     include/linux/compile.h
>   CC      kernel/latency_trace.o
> /home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/kernel/latency_trace.c:28:21: error: asm/rtc.h: No such file or directory
> make[2]: *** [kernel/latency_trace.o] Error 1
> make[1]: *** [kernel] Error 2
> make: *** [vmlinux] Error 2
> 
> Indeed, there is no include/asm-mips/rtc.h.
> 
> I commented out the include line in latency_trace.c, and everything
> compiled fine.  I'm not sure what is needed in an arch-specific rtc.h,
> but compiling without it for the mips arch caused no problems.
> Should I create a patch with a stub for rtc.h for mips?

Hm, dunno why the rtc.h include was added.

> As an aside, this has me worried.  Is anyone else doing any
> RT Preempt testing or work on MIPS platforms, or am I forging
> new ground? :-)

I fear you are the one who is in charge to get mips working again :)
But as always, there are bad news and good news:  As far as I heard
last week John Cooper is looking into this as well.

Thanks,

	tglx

Re: rt-preempt: problem compiling rt-preempt 2.6.23.1-rt11 on MIPS

[Steven Rostedt]

On Thu, 15 Nov 2007, Thomas Gleixner wrote:

> On Wed, 14 Nov 2007, Tim Bird wrote:
>
> Added Steven and John to CC
>
> > I applied the patches in patch-2.6.23.1-rt11-broken-out.tar.bz2
> > to a Linux kernel version 2.6.23.1 (along with a few other
> > board specific patches).
> >
> > I got the following compilation error:
> >
> >   GEN     /home/tbird/work/rt-preempt/build/tx49/Makefile
> >   CHK     include/linux/version.h
> >   CHK     include/linux/utsrelease.h
> >   CALL    /home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/scripts/checksyscalls.sh
> >   CHK     include/linux/compile.h
> >   CC      kernel/latency_trace.o
> > /home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/kernel/latency_trace.c:28:21: error: asm/rtc.h: No such file or directory
> > make[2]: *** [kernel/latency_trace.o] Error 1
> > make[1]: *** [kernel] Error 2
> > make: *** [vmlinux] Error 2
> >
> > Indeed, there is no include/asm-mips/rtc.h.

Looks like that may be just an artifact from older days.

I'll do some more tests, and it it truely is, I'll go and remove the added
headers.

> >
> > I commented out the include line in latency_trace.c, and everything
> > compiled fine.  I'm not sure what is needed in an arch-specific rtc.h,
> > but compiling without it for the mips arch caused no problems.
> > Should I create a patch with a stub for rtc.h for mips?
>
> Hm, dunno why the rtc.h include was added.
>
> > As an aside, this has me worried.  Is anyone else doing any
> > RT Preempt testing or work on MIPS platforms, or am I forging
> > new ground? :-)
>
> I fear you are the one who is in charge to get mips working again :)
> But as always, there are bad news and good news:  As far as I heard
> last week John Cooper is looking into this as well.


I'm just about to release 2.6.24-rc2-rt1 and I'm sure mips as well as
powrepc is badly broken.  Any help in getting these back up and working
would be greatly appreciated.

-- Steve

Re: rt-preempt: problem compiling rt-preempt 2.6.23.1-rt11 on MIPS

[Tim Bird]

Steven Rostedt wrote:
> Looks like that may be just an artifact from older days.
> 
> I'll do some more tests, and it it truely is, I'll go and remove the added
> headers.

Thanks.

> I'm just about to release 2.6.24-rc2-rt1 and I'm sure mips as well as
> powrepc is badly broken.  Any help in getting these back up and working
> would be greatly appreciated.

I'll probably have some pretty basic questions.
If you don't mind an RT newbie helping, I'll do what I can. :-)

 -- Tim

BTW - I was just trying to cross-compile the IBM test programs
(rt-test-0.6) today, and had some problems.  I didn't want to
bug anyone until I took a closer look at it, but would it be
best to report problems with that here on LKML, on
linux-rt-users, or just directly to Darren Hart?
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: rt-preempt: problem compiling rt-preempt 2.6.23.1-rt11 on MIPS

[Steven Rostedt]

On Wed, 14 Nov 2007, Tim Bird wrote:

> > I'm just about to release 2.6.24-rc2-rt1 and I'm sure mips as well as
> > powrepc is badly broken.  Any help in getting these back up and working
> > would be greatly appreciated.
>
> I'll probably have some pretty basic questions.
> If you don't mind an RT newbie helping, I'll do what I can. :-)

We can use all the help we can get :-)

> BTW - I was just trying to cross-compile the IBM test programs
> (rt-test-0.6) today, and had some problems.  I didn't want to
> bug anyone until I took a closer look at it, but would it be
> best to report problems with that here on LKML, on
> linux-rt-users, or just directly to Darren Hart?


Things related to rt-test should go to Darren Hart and to linux-rt-users.
We only CC LKML on RT kernel issues.

-- Steve

Re: rt-preempt: problem compiling rt-preempt 2.6.23.1-rt11 on MIPS

[john cooper]

Thomas Gleixner wrote:

> I fear you are the one who is in charge to get mips working again :)
> But as always, there are bad news and good news:  As far as I heard
> last week John Cooper is looking into this as well.

I'm not actively working on it but AFAIK I may have been
the last one to touch it when I did the mips version back
at Timesys.  Although I was able to get a functional port
of the work there were gremlins I never had sufficient
time to address.  That is a relatively minor issue.

The more daunting problem stems from limitations in the MIPS
ABI which makes the latency trace support problematic.
Rather than rehash the issue:

     http://lists.linuxcoding.com/kernel/2005-q4/msg10163.html

Until we have a usable instrumentation solution in place,
characterization, debug, and support of PREEMPT_RT for MIPS
is going to be a challenge.

-john

-- 
john.cooper@third-harmonic.com

MIPS RT debug support

[Tim Bird]

john cooper wrote:
> The more daunting problem stems from limitations in the MIPS
> ABI which makes the latency trace support problematic.
> Rather than rehash the issue:
> 
>     http://lists.linuxcoding.com/kernel/2005-q4/msg10163.html
> 
> Until we have a usable instrumentation solution in place,
> characterization, debug, and support of PREEMPT_RT for MIPS
> is going to be a challenge.

Agreed.  I have been using KFT (Kernel Function Trace)
on MIPS, and it has decent support for function traceback
reporting, but it's not currently integrated with latency-trace
at all.   We should discuss if this could possibly be
used to debug RT-preempt.  It is much heavier weight than
the mcount stuff, but uses similar (but not identical)
gcc profiling instrumentation.  I'm not sure if the
two can be turned on together, or how hard it would
be to move latency-trace onto -finstrument_functions.

But it's probably worth researching a little.  We'll
need something to give insight into the problem paths.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: MIPS RT debug support

[Steven Rostedt]

On Thu, 15 Nov 2007, Tim Bird wrote:

> john cooper wrote:
> > The more daunting problem stems from limitations in the MIPS
> > ABI which makes the latency trace support problematic.
> > Rather than rehash the issue:
> >
> >     http://lists.linuxcoding.com/kernel/2005-q4/msg10163.html
> >
> > Until we have a usable instrumentation solution in place,
> > characterization, debug, and support of PREEMPT_RT for MIPS
> > is going to be a challenge.
>
> Agreed.  I have been using KFT (Kernel Function Trace)
> on MIPS, and it has decent support for function traceback
> reporting, but it's not currently integrated with latency-trace
> at all.   We should discuss if this could possibly be
> used to debug RT-preempt.  It is much heavier weight than
> the mcount stuff, but uses similar (but not identical)
> gcc profiling instrumentation.  I'm not sure if the
> two can be turned on together, or how hard it would
> be to move latency-trace onto -finstrument_functions.

I'm not familiar with the KFT but I'm sure it would be easy to port
latency_trace to it. Really, all the mcount does is make a wrapper to pass
to the trace calls.

Here's the code for mcount in arch/x86/kernel/entry_64.S:

ENTRY(mcount)
	cmpl $0, mcount_enabled
	jz out

	push %rbp
	mov %rsp,%rbp

	push %r11
	push %r10
	push %r9
	push %r8
	push %rdi
	push %rsi
	push %rdx
	push %rcx
	push %rax

	mov 0x0(%rbp),%rax
	mov 0x8(%rbp),%rdi
	mov 0x8(%rax),%rsi

	call   __trace

	pop %rax
	pop %rcx
	pop %rdx
	pop %rsi
	pop %rdi
	pop %r8
	pop %r9
	pop %r10
	pop %r11

	pop %rbp
out:
	ret


Which simply passes to __trace the rip that jumped here, and (if possible)
the rip of that caller. The parent rip is not necessary.

 >
> But it's probably worth researching a little.  We'll
> need something to give insight into the problem paths.

If the KFT could do the above, it should be trivial to adapt.

Hmm, if someone is willing to send me a free mips box, I may do it myself
;-)

-- Steve

Re: MIPS RT debug support

[john cooper]

Steven Rostedt wrote:
> On Thu, 15 Nov 2007, Tim Bird wrote:
> 
>> john cooper wrote:
>>> The more daunting problem stems from limitations in the MIPS
>>> ABI which makes the latency trace support problematic.
>>> Rather than rehash the issue:
>>>
>>>     http://lists.linuxcoding.com/kernel/2005-q4/msg10163.html
>>>
>>> Until we have a usable instrumentation solution in place,
>>> characterization, debug, and support of PREEMPT_RT for MIPS
>>> is going to be a challenge.
>> Agreed.  I have been using KFT (Kernel Function Trace)
>> on MIPS, and it has decent support for function traceback
>> reporting, but it's not currently integrated with latency-trace
>> at all.   We should discuss if this could possibly be
>> used to debug RT-preempt.  It is much heavier weight than
>> the mcount stuff, but uses similar (but not identical)
>> gcc profiling instrumentation.  I'm not sure if the
>> two can be turned on together, or how hard it would
>> be to move latency-trace onto -finstrument_functions.
> 
> I'm not familiar with the KFT but I'm sure it would be easy to port
> latency_trace to it. Really, all the mcount does is make a wrapper to pass
> to the trace calls.

It isn't an issue of getting a hook into the FUNCTION_PROLOGUE
(mcount() here) but rather of emulating the CALLER_ADDR[0123]
defs which map onto the gcc internal __builtin_return_address().
Doing so using the affectionately dubbed "Three Stooges Algorithm"
called out in the MIPS ABI is both complex and nondeterministic.

The entry FUNCTION_PROLOGUE hook provides a means to log the path
traveled thus far.  __builtin_return_address() gives a way to
snapshot a stub of the stack invocation. Together they provide
somewhat complimentary but useful debug information.

We could log the stack invocation progress in the latency
instrumentation itself by noting a new invocation in the
FUNCTION_PROLOGUE and unwind of the same in the currently
unused FUNCTION_EPILOGUE hook.  So we're just shadowing the
actual runtime stack in effect with a data structure which
can be traversed far more easily.

-john

-- 
john.cooper@third-harmonic.com

Re: MIPS RT debug support

[Steven Rostedt]

On Thu, 15 Nov 2007, john cooper wrote:

>
> It isn't an issue of getting a hook into the FUNCTION_PROLOGUE
> (mcount() here) but rather of emulating the CALLER_ADDR[0123]
> defs which map onto the gcc internal __builtin_return_address().
> Doing so using the affectionately dubbed "Three Stooges Algorithm"
> called out in the MIPS ABI is both complex and nondeterministic.
>
> The entry FUNCTION_PROLOGUE hook provides a means to log the path
> traveled thus far.  __builtin_return_address() gives a way to
> snapshot a stub of the stack invocation. Together they provide
> somewhat complimentary but useful debug information.
>
> We could log the stack invocation progress in the latency
> instrumentation itself by noting a new invocation in the
> FUNCTION_PROLOGUE and unwind of the same in the currently
> unused FUNCTION_EPILOGUE hook.  So we're just shadowing the
> actual runtime stack in effect with a data structure which
> can be traversed far more easily.
>

IIRC, only CALLER_ADDR0 is actually used (I've found that the others are
mostly unreliable).  I thought that mips always has one register that
holds the return address of the function. It's been several years since
I've worked on mips (I'd love to do it again, if someone sends me a box!,
hint hint).  So the __builtin_return_address(0) should simply return that
register. Now I'm being a bit lazy, and I'm not pulling out my MIPS books,
so I could be completely wrong on this.

-- Steve

Re: MIPS RT debug support

[john cooper]

Steven Rostedt wrote:
> IIRC, only CALLER_ADDR0 is actually used (I've found that the others are
> mostly unreliable).

I seem to recall more use was made of __builtin_return_address(n)
for 0 < n in the past compared to the current code.  Likely for
shallow frames the 0 < n calls were potentially returning invalid
data which is why they currently appear to be commented out.

Rather than lobbying to fix __builtin_return_address(), replacing
it with a primitive which works in all cases seems an easier
solution.  That stub stack frame really does provide useful debug
information.  In the case of ARM I simply walked the stack.  Worst
case in that vintage of the code was four iterations so the overhead
was in the noise.  That same solution could easily be applied to
other architectures.

> It's been several years since
> I've worked on mips (I'd love to do it again, if someone sends me a box!,
> hint hint).

Concerning a MIPS board, if no freebies should arrive you
might want to dig up a linux footprint friendly version of
a Linksys WRT54GS.

-john

-- 
john.cooper@third-harmonic.com