Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Namhyung Kim <namhyung@kernel.org>]

Hi Steve,

On Tue, Jun 28, 2016 at 09:26:52PM -0400, Steven Rostedt wrote:
> On Wed, 29 Jun 2016 09:57:41 +0900
> Minchan Kim <minchan@kernel.org> wrote:
> 
> > Hello,
> > 
> > On Tue, Jun 28, 2016 at 06:46:34PM +0200, Rabin Vincent wrote:
> > > On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:  
> > > > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:  
> > > > > I'm running some guest machines for kernel development.  For debugging
> > > > > purpose, I use lots of trace_printk() since it's faster than normal
> > > > > printk().  When kernel crash happens the trace buffer is printed on
> > > > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > > > don't want to reduce the size of ring buffer as I want to collect the
> > > > > debug info as much as possible.  And I also want to see trace from all
> > > > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > > > >
> > > > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > > > importantly, I don't want to spend the memory for the crashkernel.  
> > > 
> > > Assuming you're using QEMU:
> > > 
> > > QEMU has a dump-guest-memory command which can be used to dump the
> > > guest's entire memory to an ELF which can be loaded by the crash utility
> > > to extract the trace buffer.  This doesn't require kexec/kdump or any
> > > other support from the guest kernel.  
> > 
> > Thanks for the hint. It's surely handy rather than kexec/kdump.
> > 
> > A question is that it's possible to capture guest's entire memory
> > when guest kernel is oops?
> > I mean I don't want to capture alive guest but get snapshot image
> > when guest kernel encounters BUG_ON and see event trace from the
> > image.
> > 
> > Anyway, I tried crashtool and load trace.so but failed to load
> > extension module 'trace.so' because read_string failed in
> > ftrace_get_event_type_name of trace.c.
> > Does it work with recent kernel?
> > 
> > My kernel is 4.7.0-rc4-mm1.
> 
> It probably needs another update. I usually send patches to David
> Anderson for updates. Fujitsu started that work and was maintaining it
> for a while, but I don't think they are anymore. I have no problem
> maintaining the trace.so module.
> 
> If I get time tomorrow, I'll see if I can get it up to date again.

It seems that commit dcb0b5575d24 ("tracing: Remove
TRACE_EVENT_FL_USE_CALL_FILTER logic") changed the bit index so
it makes checking TRACE_EVENT_FL_TRACEPOINT flag failed.  It should
0x20 for newer kernels..

Anyway, this kind of problem can happen at anytime.  One needs to
update the crashtool if some struct or variable name changed.  Maybe
it'd make sense to move the crashtool into the kernel tree?

Thanks,
Namhyung