MIPS Function Tracer question

MIPS Function Tracer question

[Alan Cooper]

I've been doing some testing of the MIPS Function Tracer functionality
on the 3.3 kernel. I was surprised to find that the option to generate
frame pointers was required for tracing. When I don't enable
FRAME_POINTER along with FUNCTION_TRACER, the kernel hangs on boot. I
also noticed that a checkin to the 3.4 kernel
(b732d439cb43336cd6d7e804ecb2c81193ef63b0) no longer forces on
FRAME_POINTER when FUNCTION_TRACER is selected. I was wondering how it
works in 3.4 and beyond, so I built a Malta kernel from the latest
MIPS tree with FUNCTION_TRACING enabled and tested it with QEMU. The
kernel hung the same way. I can think of 2 reasons for this:
1. Function tracing is broken for MIPS in 3.4 and beyond.
2. The 4.5.3 GNU C compiler I'm using is generating different code for
function tracing.
I was wondering if anyone has MIPS function tracing working in 3.4 or later?

I did figure out why it's hanging and I have some changes that will
allow the function tracer to run without frame pointers, but before I
proceed I want to rule out compiler differences.

Thanks

Re: MIPS Function Tracer question

[David Daney]

On 11/29/2012 01:04 PM, Alan Cooper wrote:
> I've been doing some testing of the MIPS Function Tracer functionality
> on the 3.3 kernel. I was surprised to find that the option to generate
> frame pointers was required for tracing.

It is not really required for MIPS function tracing, but the Kconfigs 
for some reason set it.

>  When I don't enable
> FRAME_POINTER along with FUNCTION_TRACER, the kernel hangs on boot. I
> also noticed that a checkin to the 3.4 kernel
> (b732d439cb43336cd6d7e804ecb2c81193ef63b0) no longer forces on
> FRAME_POINTER when FUNCTION_TRACER is selected. I was wondering how it
> works in 3.4 and beyond, so I built a Malta kernel from the latest
> MIPS tree with FUNCTION_TRACING enabled and tested it with QEMU. The
> kernel hung the same way. I can think of 2 reasons for this:
> 1. Function tracing is broken for MIPS in 3.4 and beyond.
> 2. The 4.5.3 GNU C compiler I'm using is generating different code for
> function tracing.

Function tracing works best with recent versions of GCC (those that 
support -mmcount-ra-address).

> I was wondering if anyone has MIPS function tracing working in 3.4 or later?

Yes.  Using GCC 4.7.0 on an octeon kernel (based on 3.4.14):

# tracer: function_graph
#
# CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
   1)               |  __fsnotify_parent() {
   1)   7.154 us    |  } /* __fsnotify_parent */
   1)               |  fsnotify() {
   1)               |    __srcu_read_lock() {
   1)               |      add_preempt_count() {
   1)   1.356 us    |      } /* add_preempt_count */
   1)               |      sub_preempt_count() {
   1)   1.385 us    |      } /* sub_preempt_count */
   1)   6.747 us    |    } /* __srcu_read_lock */
   1)               |    __srcu_read_unlock() {
   1)               |      add_preempt_count() {
   1)   1.383 us    |      } /* add_preempt_count */
   1)               |      sub_preempt_count() {
   1)   1.358 us    |      } /* sub_preempt_count */
   1)   6.642 us    |    } /* __srcu_read_unlock */
   1) + 17.861 us   |  } /* fsnotify */
.
.
.



>
> I did figure out why it's hanging and I have some changes that will
> allow the function tracer to run without frame pointers, but before I
> proceed I want to rule out compiler differences.
>
> Thanks
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>

Re: MIPS Function Tracer question

[David Daney]

On 11/30/2012 06:53 AM, Alan Cooper wrote:
> I needed to be more specific. The issues I'm seeing are only on 32BIT
> platforms with dynamic tracing.
> Let me explain the issue. When the compiler flag "-pg" is specified
> for 32BIT platforms to enable tracing, the compiler adds "addiu
> sp,sp,-8" in the delay slot after every call to _mcount (some legacy
> thing where 2 arguments used to be pushed on the stack). The _mcount
> routine is expected to adjust the sp by 8 before returning.

If the kernel's tracer infrastructure is broken for 32-bit kernels, then 
it should be fixed.  I think it was only really tested on 64-bit kernels.

It seems that the sp adjustment should be replaced by NOP as well if 
tracing is disabled.

> The
> problem is when tracing is disabled, all calls to _mcount are
> dynamically converted from "jal _mcount" to "nop" but the following
> "addiu sp,sp,-8" instruction is unchanged and when executed leaves the
> sp off by -8 for the remainder of the function. This bug is hidden
> when the compiler is told to use frame pointers because all offsets
> into the stack use the fp instead of the sp and the sp is restored
> from the frame pointer instead of just adding a constant to sp. When
> frame pointers are not enabled the code crashes. I don't think the
> toolchain version makes any difference because the _mcount assembly
> language routine adjusts the stack pointer by 8 if not CONFIG_64BIT
> regardless of the toolchain version.
>
> On Thu, Nov 29, 2012 at 6:00 PM, David Daney <ddaney.cavm@gmail.com> wrote:
>> On 11/29/2012 01:04 PM, Alan Cooper wrote:
>>>
>>> I've been doing some testing of the MIPS Function Tracer functionality
>>> on the 3.3 kernel. I was surprised to find that the option to generate
>>> frame pointers was required for tracing.
>>
>>
>> It is not really required for MIPS function tracing, but the Kconfigs for
>> some reason set it.
>>
>>
>>>   When I don't enable
>>> FRAME_POINTER along with FUNCTION_TRACER, the kernel hangs on boot. I
>>> also noticed that a checkin to the 3.4 kernel
>>> (b732d439cb43336cd6d7e804ecb2c81193ef63b0) no longer forces on
>>> FRAME_POINTER when FUNCTION_TRACER is selected. I was wondering how it
>>> works in 3.4 and beyond, so I built a Malta kernel from the latest
>>> MIPS tree with FUNCTION_TRACING enabled and tested it with QEMU. The
>>> kernel hung the same way. I can think of 2 reasons for this:
>>> 1. Function tracing is broken for MIPS in 3.4 and beyond.
>>> 2. The 4.5.3 GNU C compiler I'm using is generating different code for
>>> function tracing.
>>
>>
>> Function tracing works best with recent versions of GCC (those that support
>> -mmcount-ra-address).
>>
>>
>>> I was wondering if anyone has MIPS function tracing working in 3.4 or
>>> later?
>>
>>
>> Yes.  Using GCC 4.7.0 on an octeon kernel (based on 3.4.14):
>>
>> # tracer: function_graph
>> #
>> # CPU  DURATION                  FUNCTION CALLS
>> # |     |   |                     |   |   |   |
>>    1)               |  __fsnotify_parent() {
>>    1)   7.154 us    |  } /* __fsnotify_parent */
>>    1)               |  fsnotify() {
>>    1)               |    __srcu_read_lock() {
>>    1)               |      add_preempt_count() {
>>    1)   1.356 us    |      } /* add_preempt_count */
>>    1)               |      sub_preempt_count() {
>>    1)   1.385 us    |      } /* sub_preempt_count */
>>    1)   6.747 us    |    } /* __srcu_read_lock */
>>    1)               |    __srcu_read_unlock() {
>>    1)               |      add_preempt_count() {
>>    1)   1.383 us    |      } /* add_preempt_count */
>>    1)               |      sub_preempt_count() {
>>    1)   1.358 us    |      } /* sub_preempt_count */
>>    1)   6.642 us    |    } /* __srcu_read_unlock */
>>    1) + 17.861 us   |  } /* fsnotify */
>> .
>> .
>> .
>>
>>
>>
>>>
>>> I did figure out why it's hanging and I have some changes that will
>>> allow the function tracer to run without frame pointers, but before I
>>> proceed I want to rule out compiler differences.
>>>
>>> Thanks
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>
>>>
>>

Re: MIPS Function Tracer question

[Steven Rostedt]

On Thu, 2012-11-29 at 15:00 -0800, David Daney wrote:
> On 11/29/2012 01:04 PM, Alan Cooper wrote:
> > I've been doing some testing of the MIPS Function Tracer functionality
> > on the 3.3 kernel. I was surprised to find that the option to generate
> > frame pointers was required for tracing.
> 
> It is not really required for MIPS function tracing, but the Kconfigs 
> for some reason set it.
> 

The issue is with x86. Gcc wont compile if you have -pg and
-fomit-frame-pointer on x86. I originally forced function tracing to
select FRAME_POINTER, but because now on x86 with -mfentry, -pg no
longer requires frame pointers being set, I just let -pg complain one
way or the other. I believe that gcc by default will not add frame
pointers. Thus adding function tracing just prevents
-fomit-frame-pointer from being set, and if -pg requires frame pointers
it will automatically enable them otherwise they should not be enabled.

-- Steve

> >  When I don't enable
> > FRAME_POINTER along with FUNCTION_TRACER, the kernel hangs on boot. I
> > also noticed that a checkin to the 3.4 kernel
> > (b732d439cb43336cd6d7e804ecb2c81193ef63b0) no longer forces on
> > FRAME_POINTER when FUNCTION_TRACER is selected. I was wondering how it
> > works in 3.4 and beyond, so I built a Malta kernel from the latest
> > MIPS tree with FUNCTION_TRACING enabled and tested it with QEMU. The
> > kernel hung the same way. I can think of 2 reasons for this:
> > 1. Function tracing is broken for MIPS in 3.4 and beyond.
> > 2. The 4.5.3 GNU C compiler I'm using is generating different code for
> > function tracing.
> 

Re: MIPS Function Tracer question

[Ralf Baechle]

On Mon, Dec 03, 2012 at 09:40:48AM -0500, Steven Rostedt wrote:

> The issue is with x86. Gcc wont compile if you have -pg and
> -fomit-frame-pointer on x86. I originally forced function tracing to
> select FRAME_POINTER, but because now on x86 with -mfentry, -pg no
> longer requires frame pointers being set, I just let -pg complain one
> way or the other. I believe that gcc by default will not add frame
> pointers. Thus adding function tracing just prevents
> -fomit-frame-pointer from being set, and if -pg requires frame pointers
> it will automatically enable them otherwise they should not be enabled.

On architectures such as MIPS where a frame pointer is not required for
debugging -O and higher imply -fomit-frame-pointer.

  Ralf

Re: MIPS Function Tracer question

[Steven Rostedt]

On Mon, 2012-12-03 at 17:13 +0100, Ralf Baechle wrote:
> On Mon, Dec 03, 2012 at 09:40:48AM -0500, Steven Rostedt wrote:
> 
> > The issue is with x86. Gcc wont compile if you have -pg and
> > -fomit-frame-pointer on x86. I originally forced function tracing to
> > select FRAME_POINTER, but because now on x86 with -mfentry, -pg no
> > longer requires frame pointers being set, I just let -pg complain one
> > way or the other. I believe that gcc by default will not add frame
> > pointers. Thus adding function tracing just prevents
> > -fomit-frame-pointer from being set, and if -pg requires frame pointers
> > it will automatically enable them otherwise they should not be enabled.
> 
> On architectures such as MIPS where a frame pointer is not required for
> debugging -O and higher imply -fomit-frame-pointer.

Then this shouldn't be an issue for MIPS.

-- Steve