Re: 2.6.33: ftrace triggers soft lockup

[Frederic Weisbecker <fweisbec@gmail.com>]

On Fri, Mar 05, 2010 at 10:06:43AM -0500, Steven Rostedt wrote:
> On Fri, 2010-03-05 at 15:16 +0800, Américo Wang wrote:
> > On Fri, Mar 5, 2010 at 12:14 PM, Américo Wang <xiyou.wangcong@gmail.com> wrote:
> > > On Thu, Mar 4, 2010 at 9:54 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> > >> On Wed, 2010-03-03 at 14:04 +0800, Américo Wang wrote:
> > >>> I am not sure if this is ftrace's fault, but it is ftrace who triggers
> > >>> the soft lockup. On my machine, it is pretty easy, just run:
> > >>>
> > >>> echo function_graph > current_tracer
> > >>>
> 
> > >
> > > I can't say that because I didn't try -rc6.
> > >
> > 
> > Sigh, 2.6.33-rc6 doesn't work, even 2.6.32 doesn't work...
> 
> So basically you are saying that the function_graph tracer, when enabled
> has a high overhead? Well, unfortunately, that's expected.
> 
> The function_graph tracer traces the start and end of every function. It
> uses the same mechanism as function tracer to trace the start of the
> function (mcount), but to trace the exit of a function, in the enter of
> the function it hijacks the return address and replaces it to call a
> trampoline. This trampoline will do the trace and then jump back to the
> original return address.
> 
> Doing this breaks branch prediction in the CPU, as the CPU uses call/ret
> as part of its branch prediction analysis. So function graph tracing is
> not just twice as slow as function tracing, it actually has a bigger
> impact than that.


It's true it has a high overhead, but not to the point of
making the whole system unusable. We are supposed to be even
far from that. I'm currently able to turn on the function graph
tracer and use firefox without problems. It's just a bit slower
but it's far from a visible starvation.

And Li seems to see the same thing.
For now I can not test, but I will try this week-end.