Re: trace_printk() support in trace-cmd

[Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>]

(2010/12/16 19:20), Avi Kivity wrote:
> On 12/13/2010 01:20 PM, Masami Hiramatsu wrote:
>> (2010/12/13 2:47), Avi Kivity wrote:
>> >  On 12/12/2010 07:43 PM, Arnaldo Carvalho de Melo wrote:
>> >>  Em Sun, Dec 12, 2010 at 07:42:06PM +0200, Avi Kivity escreveu:
>> >>  >   On 12/12/2010 07:36 PM, Arnaldo Carvalho de Melo wrote:
>> >>  >   >Em Sun, Dec 12, 2010 at 06:35:24PM +0200, Avi Kivity escreveu:
>> >>  >   >>    On 11/23/2010 05:45 PM, Steven Rostedt wrote:
>> >>  >   >>    >Again, the work around is to replace your trace_printks() with
>> >>  >   >>    >__trace_printk(_THIS_IP_, ...) or just modify the trace_printk() macro
>> >>  >   >>    >in include/linux/kernel.h to always use the __trace_printk() version.
>> >>  >   >>
>> >>  >   >>    This works; I'm using it for now (I tried to use 'perf probe', but I
>> >>  >   >>    get unpredictable results, like null pointer derefs).
>> >>  >   >
>> >>  >   >Can you tell us which functions, environment, etc?
>> >>  >
>> >>  >   Something around 2.6.27-rc4; example functions are FNAME(fetch) in
>> >>  >   arch/x86/kvm/paging_tmpl.h; compiled modular (which was Steven's
>> >>  >   guess as to why it fails).
>> >>  >
>> >>  >   (note, the failure is with trace-cmd, not /sys/kernel/debug/tracing).
>> >>
>> >>  I mean the "I tried to use 'perf probe'" part.
>> >
>> >  Well, same, more or less.
>> >
>> >    perf probe -m kvm --add 'fetch_access=paging64_fetch pt_access=gw->pt_access pte_access=gw->pte_access dirty'
>> >
>> >  would return garbage for gw->*, and the log would show the exception handler called.  gw is most certainly valid.
>> >
>>
>> Thank you for reporting.
>> Hmm, actually, pagefaults could happen on fetching variables. But
>> fetching argument routines should handle it...
> 
> They did handle it (or so I understood from the logs).  But they shouldn't have 
> occured in the first place, since gw was dereferenceable (and the function dereferences it).

Ah, OK. Sometimes, it's hard to find the register/memory location of
local variables. (and sometimes it fails)

>  So something went wrong while fetching gw itself (do you interpret the
> dwarf tables to find where the variable is stored?)

Hm, yes, you can use eu-readelf to dump debuginfo, and also objdump will help you
to find the address and assembler code.

> 
>> I'd like to check it, could you tell me details? for example, that exception log,
>> kprobe-tracer's event definition(you can see it via debugfs/tracing/kprobe-events)
>> and the result of `perf probe -L paging64_fetch:0-10`.
> 
> I no longer have the logs, I'll try to reproduce it later.

Oh, Thank you! :)


-- 
Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com