Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Minchan Kim <minchan@kernel.org>]

Hello,

On Tue, Jun 28, 2016 at 06:46:34PM +0200, Rabin Vincent wrote:
> On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:
> > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> > > I'm running some guest machines for kernel development.  For debugging
> > > purpose, I use lots of trace_printk() since it's faster than normal
> > > printk().  When kernel crash happens the trace buffer is printed on
> > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > don't want to reduce the size of ring buffer as I want to collect the
> > > debug info as much as possible.  And I also want to see trace from all
> > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > >
> > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > importantly, I don't want to spend the memory for the crashkernel.
> 
> Assuming you're using QEMU:
> 
> QEMU has a dump-guest-memory command which can be used to dump the
> guest's entire memory to an ELF which can be loaded by the crash utility
> to extract the trace buffer.  This doesn't require kexec/kdump or any
> other support from the guest kernel.

Thanks for the hint. It's surely handy rather than kexec/kdump.

A question is that it's possible to capture guest's entire memory
when guest kernel is oops?
I mean I don't want to capture alive guest but get snapshot image
when guest kernel encounters BUG_ON and see event trace from the
image.

Anyway, I tried crashtool and load trace.so but failed to load
extension module 'trace.so' because read_string failed in
ftrace_get_event_type_name of trace.c.
Does it work with recent kernel?

My kernel is 4.7.0-rc4-mm1.

> 
> It's apparently even possible to run QEMU with the guest memory in a
> file and load that to crash directly, although this is not something
> I've had a chance to try out myself:
> 
> https://github.com/crash-utility/crash/commit/89ed9d0a7f7da4578294a492c1ad857244ce7352