Bug with jprobes and function graph tracer

Bug with jprobes and function graph tracer

[Steven Rostedt]

Hi Masami,

First, I would like to wish you a Happy New Year :-)

OK, now back to business.


It has come to my attention that jprobes and function graph tracing do
not play nicely together. I think you once told me they have issues as
well.

A simple way to test this is to load the jprobe_example.ko module and
then run function graph tracing.

Should we try to fix this? Or should it just be that users need to know
not to enable the two together. Or we can prevent one from starting if
the other is being used.

Looking forward to hearing your thoughts on the subject.

-- Steve

Re: Bug with jprobes and function graph tracer

[Masami Hiramatsu]

(2015/01/09 2:20), Steven Rostedt wrote:
> Hi Masami,
> 
> First, I would like to wish you a Happy New Year :-)

Happy New Year! :)

> OK, now back to business.
> 
> 
> It has come to my attention that jprobes and function graph tracing do
> not play nicely together. I think you once told me they have issues as
> well.
> 
> A simple way to test this is to load the jprobe_example.ko module and
> then run function graph tracing.

I see. Jprobes will mess up the function call path.

> Should we try to fix this? Or should it just be that users need to know
> not to enable the two together. Or we can prevent one from starting if
> the other is being used.

Yeah we should try to fix this, but not so high priority since jprobe
is not used so frequently. I guess one easy solution is adding notrace
to all jprobe handlers for hiding it from function-graph tracer.
If there is a way to check a function is notrace or not, I can use that
in register_jprobe and reject it if the handler is not "notrace"ed. :)

Thank you,

> 
> Looking forward to hearing your thoughts on the subject.
> 
> -- Steve
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Bug with jprobes and function graph tracer

[Steven Rostedt]

On Tue, 13 Jan 2015 20:39:33 +0900
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:

> 
> Yeah we should try to fix this, but not so high priority since jprobe
> is not used so frequently. I guess one easy solution is adding notrace
> to all jprobe handlers for hiding it from function-graph tracer.
> If there is a way to check a function is notrace or not, I can use that
> in register_jprobe and reject it if the handler is not "notrace"ed. :)

Adding notrace should be the last resort. Every notrace in the kernel
makes function tracing a little more useless. It bothers me even more
when notrace is added because of function graph tracing but the
function still works for function tracing.

I worked on this for the last few days and have a patch that fixes the
problem. I'll be posting it later today. It also uncovered another
unrelated bug that I worked on yesterday, otherwise I would have posted
these fixes already.

I have a fix for the other bug now but it needs to be cleaned up
(remove all the debugging prints from it). When that's complete I'll
post the patches and get them ready for mainline and stable.

Thanks,

-- Steve

Re: Bug with jprobes and function graph tracer

[Masami Hiramatsu]

(2015/01/13 21:22), Steven Rostedt wrote:
> On Tue, 13 Jan 2015 20:39:33 +0900
> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> 
>>
>> Yeah we should try to fix this, but not so high priority since jprobe
>> is not used so frequently. I guess one easy solution is adding notrace
>> to all jprobe handlers for hiding it from function-graph tracer.
>> If there is a way to check a function is notrace or not, I can use that
>> in register_jprobe and reject it if the handler is not "notrace"ed. :)
> 
> Adding notrace should be the last resort. Every notrace in the kernel
> makes function tracing a little more useless. It bothers me even more
> when notrace is added because of function graph tracing but the
> function still works for function tracing.

I see.

> 
> I worked on this for the last few days and have a patch that fixes the
> problem. I'll be posting it later today. It also uncovered another
> unrelated bug that I worked on yesterday, otherwise I would have posted
> these fixes already.

Great! I look forward that! :)

> 
> I have a fix for the other bug now but it needs to be cleaned up
> (remove all the debugging prints from it). When that's complete I'll
> post the patches and get them ready for mainline and stable.

Thank you,

> 
> Thanks,
> 
> -- Steve
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Bug with jprobes and function graph tracer

[Steven Rostedt]

On Tue, 13 Jan 2015 21:46:33 +0900
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:


> > 
> > I worked on this for the last few days and have a patch that fixes the
> > problem. I'll be posting it later today. It also uncovered another
> > unrelated bug that I worked on yesterday, otherwise I would have posted
> > these fixes already.
> 
> Great! I look forward that! :)

I was all set to send the patches but during testing found that this
isn't just affected by the fentry code, but also happens when jprobes
and kprobes uses the breakpoint method (old mcount and i386).

I guess this has always been broken from when jprobes and function
graph tracing were together.

Anyway, I think I have a simpler solution which simulates making the
jprobes "notrace". At least this solution will allow us to have jprobe
callbacks traced in the future. But for now, instead of crashing, we'll
just disable function graph tracing while its in a jprobe.

I'll see if I can get that patch working today (I should stop saying
that).

But all is not lost. I found two bugs in the current implementation of
the function graph tracer because of this :-)

-- Steve