Re: [ath9k-devel] ath9k: massive unexplained latency in 2.6.27 (rc5, rc6, probably others)

[Ingo Molnar <mingo@elte.hu>]

* Steven Noonan <steven@uplinklabs.net> wrote:

> On Fri, Sep 19, 2008 at 1:17 AM, Ingo Molnar <mingo@elte.hu> wrote:
> > in tip/master there's an ftrace_printk() facility. You can just replace
> > the verbose printk()s with ftrace_printk() and see the result in
> > /debug/tracing/trace.
> >
> > This sort of tracing has very low overhead and can be used in an easy
> > ad-hoc manner with no extra infrastructure. Here's a few quick-start
> > links about how to enable the scheduler tracer:
> >
> >  http://redhat.com/~mingo/sched-devel.git/readme-tracer.txt
> >  http://redhat.com/~mingo/sched-devel.git/howto-trace-latencies.txt
> >
> > any ftrace_printk() you add in the kernel will show up in that trace.
> >
> > (If the scheduling events are uninteresting and clutter the output then
> > you might want to remove the scheduler tracing entries from kernel/*.c
> > by removing the trace_sched_*() calls or use a less noisy tracer.)
> >
> > to get enough of a trace history you might want to increase the number
> > of trace entries in /debug/tracing/trace_entries from 16K to 128K or so.
> >
> 
> Thanks for the suggestion, I'll do that. :)

one more suggestion: you should first check how large the 'scope' of 
your trace is - i.e. how long of a time span it can cover, in the 
workload that you are going to trace.

if it's too short - say just a few hundred milliseconds then you need a 
larger trace buffer or fewer trace entries.

another thing: you can auto-stop tracing on the failure condition. That 
way you dont have to worry about whether you notice the badness. 
Unfortunately there's no easy callback to just anonymously stop all 
tracing that goes on. There's ftrace_kill_atomic() but i'm not sure it 
properly stops the scheduler tracer.

What is needed is to set kernel/trace/trace.c's tracing_disabled 
variable to 0. I've Cc:-ed Steve Rostedt - Steve, why is there no easy 
ad-hoc way to trigger a stop-tracing event without elaborate changes?

	Ingo