[QUESTION] Is there a better way to get ftrace dump on guest?

[QUESTION] Is there a better way to get ftrace dump on guest?

[Namhyung Kim]

Hello,

I'm running some guest machines for kernel development.  For debugging
purpose, I use lots of trace_printk() since it's faster than normal
printk().  When kernel crash happens the trace buffer is printed on
console (I set ftrace_dump_on_oops) but it takes too much time.  I
don't want to reduce the size of ring buffer as I want to collect the
debug info as much as possible.  And I also want to see trace from all
cpu so 'ftrace_dump_on_oop = 2' is not an option.

I know the kexec/kdump (and the crash tool) can dump and analyze the
trace buffer later.  But it's cumbersome to do it everytime and more
importantly, I don't want to spend the memory for the crashkernel.

So what is the best way to handle this?  I'd like to know how others
setup the debugging environment..

Thanks,
Namhyung

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Namhyung Kim]

Send again to correct addresses, sorry!

On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> Hello,
>
> I'm running some guest machines for kernel development.  For debugging
> purpose, I use lots of trace_printk() since it's faster than normal
> printk().  When kernel crash happens the trace buffer is printed on
> console (I set ftrace_dump_on_oops) but it takes too much time.  I
> don't want to reduce the size of ring buffer as I want to collect the
> debug info as much as possible.  And I also want to see trace from all
> cpu so 'ftrace_dump_on_oop = 2' is not an option.
>
> I know the kexec/kdump (and the crash tool) can dump and analyze the
> trace buffer later.  But it's cumbersome to do it everytime and more
> importantly, I don't want to spend the memory for the crashkernel.
>
> So what is the best way to handle this?  I'd like to know how others
> setup the debugging environment..

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Steven Rostedt]

On Tue, 28 Jun 2016 15:33:18 +0900
Namhyung Kim <namhyung@kernel.org> wrote:

> Send again to correct addresses, sorry!
> 
> On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> > Hello,
> >
> > I'm running some guest machines for kernel development.  For debugging
> > purpose, I use lots of trace_printk() since it's faster than normal
> > printk().  When kernel crash happens the trace buffer is printed on
> > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > don't want to reduce the size of ring buffer as I want to collect the
> > debug info as much as possible.  And I also want to see trace from all
> > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> >
> > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > trace buffer later.  But it's cumbersome to do it everytime and more
> > importantly, I don't want to spend the memory for the crashkernel.
> >
> > So what is the best way to handle this?  I'd like to know how others
> > setup the debugging environment..  

Heh, I'd say something helpful but you basically already shot down all
of my advice, because what I do is...

1) Reduce the size of the ring buffer

2) Dump out just one CPU

3) use kexec/kdump and make a crash kernel to extract trace.dat from


That's my debugging environment, but it looks like you want something
else.

-- Steve

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Namhyung Kim]

Hi Steve,

On Tue, Jun 28, 2016 at 09:57:27AM -0400, Steven Rostedt wrote:
> On Tue, 28 Jun 2016 15:33:18 +0900
> Namhyung Kim <namhyung@kernel.org> wrote:
> 
> > Send again to correct addresses, sorry!
> > 
> > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> > > Hello,
> > >
> > > I'm running some guest machines for kernel development.  For debugging
> > > purpose, I use lots of trace_printk() since it's faster than normal
> > > printk().  When kernel crash happens the trace buffer is printed on
> > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > don't want to reduce the size of ring buffer as I want to collect the
> > > debug info as much as possible.  And I also want to see trace from all
> > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > >
> > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > importantly, I don't want to spend the memory for the crashkernel.
> > >
> > > So what is the best way to handle this?  I'd like to know how others
> > > setup the debugging environment..  
> 
> Heh, I'd say something helpful but you basically already shot down all
> of my advice, because what I do is...
> 
> 1) Reduce the size of the ring buffer
> 
> 2) Dump out just one CPU
> 
> 3) use kexec/kdump and make a crash kernel to extract trace.dat from
> 
> 
> That's my debugging environment, but it looks like you want something
> else.

Thanks for sharing.  Yeah, I'd like to know other ways to overcome
this if possible.  Since I don't have enough knowledge about this
area, I hope others would have better idea. :)

Thanks,
Namhyung

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Namhyung Kim]

On Wed, Jun 29, 2016 at 09:52:31AM +0900, Namhyung Kim wrote:
> Hi Steve,
> 
> On Tue, Jun 28, 2016 at 09:57:27AM -0400, Steven Rostedt wrote:
> > On Tue, 28 Jun 2016 15:33:18 +0900
> > Namhyung Kim <namhyung@kernel.org> wrote:
> > 
> > > Send again to correct addresses, sorry!
> > > 
> > > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> > > > Hello,
> > > >
> > > > I'm running some guest machines for kernel development.  For debugging
> > > > purpose, I use lots of trace_printk() since it's faster than normal
> > > > printk().  When kernel crash happens the trace buffer is printed on
> > > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > > don't want to reduce the size of ring buffer as I want to collect the
> > > > debug info as much as possible.  And I also want to see trace from all
> > > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > > >
> > > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > > importantly, I don't want to spend the memory for the crashkernel.
> > > >
> > > > So what is the best way to handle this?  I'd like to know how others
> > > > setup the debugging environment..  
> > 
> > Heh, I'd say something helpful but you basically already shot down all
> > of my advice, because what I do is...
> > 
> > 1) Reduce the size of the ring buffer
> > 
> > 2) Dump out just one CPU
> > 
> > 3) use kexec/kdump and make a crash kernel to extract trace.dat from
> > 
> > 
> > That's my debugging environment, but it looks like you want something
> > else.
> 
> Thanks for sharing.  Yeah, I'd like to know other ways to overcome
> this if possible.  Since I don't have enough knowledge about this
> area, I hope others would have better idea. :)

Now I'm thinking about extending the pstore subsystem.  AFAICS it's
the best fit for my use case.  While it only supports function tracer
with a dedicated ftrace_ops now, it can be used for ftrace dump IMHO.

Does it make sense to add a virtio pstore driver and saves the dump to
files on host?

Thanks,
Namhyung

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Rabin Vincent]

On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:
> On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> > I'm running some guest machines for kernel development.  For debugging
> > purpose, I use lots of trace_printk() since it's faster than normal
> > printk().  When kernel crash happens the trace buffer is printed on
> > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > don't want to reduce the size of ring buffer as I want to collect the
> > debug info as much as possible.  And I also want to see trace from all
> > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> >
> > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > trace buffer later.  But it's cumbersome to do it everytime and more
> > importantly, I don't want to spend the memory for the crashkernel.

Assuming you're using QEMU:

QEMU has a dump-guest-memory command which can be used to dump the
guest's entire memory to an ELF which can be loaded by the crash utility
to extract the trace buffer.  This doesn't require kexec/kdump or any
other support from the guest kernel.

It's apparently even possible to run QEMU with the guest memory in a
file and load that to crash directly, although this is not something
I've had a chance to try out myself:

https://github.com/crash-utility/crash/commit/89ed9d0a7f7da4578294a492c1ad857244ce7352

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Minchan Kim]

Hello,

On Tue, Jun 28, 2016 at 06:46:34PM +0200, Rabin Vincent wrote:
> On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:
> > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> > > I'm running some guest machines for kernel development.  For debugging
> > > purpose, I use lots of trace_printk() since it's faster than normal
> > > printk().  When kernel crash happens the trace buffer is printed on
> > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > don't want to reduce the size of ring buffer as I want to collect the
> > > debug info as much as possible.  And I also want to see trace from all
> > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > >
> > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > importantly, I don't want to spend the memory for the crashkernel.
> 
> Assuming you're using QEMU:
> 
> QEMU has a dump-guest-memory command which can be used to dump the
> guest's entire memory to an ELF which can be loaded by the crash utility
> to extract the trace buffer.  This doesn't require kexec/kdump or any
> other support from the guest kernel.

Thanks for the hint. It's surely handy rather than kexec/kdump.

A question is that it's possible to capture guest's entire memory
when guest kernel is oops?
I mean I don't want to capture alive guest but get snapshot image
when guest kernel encounters BUG_ON and see event trace from the
image.

Anyway, I tried crashtool and load trace.so but failed to load
extension module 'trace.so' because read_string failed in
ftrace_get_event_type_name of trace.c.
Does it work with recent kernel?

My kernel is 4.7.0-rc4-mm1.

> 
> It's apparently even possible to run QEMU with the guest memory in a
> file and load that to crash directly, although this is not something
> I've had a chance to try out myself:
> 
> https://github.com/crash-utility/crash/commit/89ed9d0a7f7da4578294a492c1ad857244ce7352

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Steven Rostedt]

On Wed, 29 Jun 2016 09:57:41 +0900
Minchan Kim <minchan@kernel.org> wrote:

> Hello,
> 
> On Tue, Jun 28, 2016 at 06:46:34PM +0200, Rabin Vincent wrote:
> > On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:  
> > > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:  
> > > > I'm running some guest machines for kernel development.  For debugging
> > > > purpose, I use lots of trace_printk() since it's faster than normal
> > > > printk().  When kernel crash happens the trace buffer is printed on
> > > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > > don't want to reduce the size of ring buffer as I want to collect the
> > > > debug info as much as possible.  And I also want to see trace from all
> > > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > > >
> > > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > > importantly, I don't want to spend the memory for the crashkernel.  
> > 
> > Assuming you're using QEMU:
> > 
> > QEMU has a dump-guest-memory command which can be used to dump the
> > guest's entire memory to an ELF which can be loaded by the crash utility
> > to extract the trace buffer.  This doesn't require kexec/kdump or any
> > other support from the guest kernel.  
> 
> Thanks for the hint. It's surely handy rather than kexec/kdump.
> 
> A question is that it's possible to capture guest's entire memory
> when guest kernel is oops?
> I mean I don't want to capture alive guest but get snapshot image
> when guest kernel encounters BUG_ON and see event trace from the
> image.
> 
> Anyway, I tried crashtool and load trace.so but failed to load
> extension module 'trace.so' because read_string failed in
> ftrace_get_event_type_name of trace.c.
> Does it work with recent kernel?
> 
> My kernel is 4.7.0-rc4-mm1.

It probably needs another update. I usually send patches to David
Anderson for updates. Fujitsu started that work and was maintaining it
for a while, but I don't think they are anymore. I have no problem
maintaining the trace.so module.

If I get time tomorrow, I'll see if I can get it up to date again.

-- Steve


> 
> > 
> > It's apparently even possible to run QEMU with the guest memory in a
> > file and load that to crash directly, although this is not something
> > I've had a chance to try out myself:
> > 
> > https://github.com/crash-utility/crash/commit/89ed9d0a7f7da4578294a492c1ad857244ce7352  

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Namhyung Kim]

Hi Steve,

On Tue, Jun 28, 2016 at 09:26:52PM -0400, Steven Rostedt wrote:
> On Wed, 29 Jun 2016 09:57:41 +0900
> Minchan Kim <minchan@kernel.org> wrote:
> 
> > Hello,
> > 
> > On Tue, Jun 28, 2016 at 06:46:34PM +0200, Rabin Vincent wrote:
> > > On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:  
> > > > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:  
> > > > > I'm running some guest machines for kernel development.  For debugging
> > > > > purpose, I use lots of trace_printk() since it's faster than normal
> > > > > printk().  When kernel crash happens the trace buffer is printed on
> > > > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > > > don't want to reduce the size of ring buffer as I want to collect the
> > > > > debug info as much as possible.  And I also want to see trace from all
> > > > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > > > >
> > > > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > > > importantly, I don't want to spend the memory for the crashkernel.  
> > > 
> > > Assuming you're using QEMU:
> > > 
> > > QEMU has a dump-guest-memory command which can be used to dump the
> > > guest's entire memory to an ELF which can be loaded by the crash utility
> > > to extract the trace buffer.  This doesn't require kexec/kdump or any
> > > other support from the guest kernel.  
> > 
> > Thanks for the hint. It's surely handy rather than kexec/kdump.
> > 
> > A question is that it's possible to capture guest's entire memory
> > when guest kernel is oops?
> > I mean I don't want to capture alive guest but get snapshot image
> > when guest kernel encounters BUG_ON and see event trace from the
> > image.
> > 
> > Anyway, I tried crashtool and load trace.so but failed to load
> > extension module 'trace.so' because read_string failed in
> > ftrace_get_event_type_name of trace.c.
> > Does it work with recent kernel?
> > 
> > My kernel is 4.7.0-rc4-mm1.
> 
> It probably needs another update. I usually send patches to David
> Anderson for updates. Fujitsu started that work and was maintaining it
> for a while, but I don't think they are anymore. I have no problem
> maintaining the trace.so module.
> 
> If I get time tomorrow, I'll see if I can get it up to date again.

It seems that commit dcb0b5575d24 ("tracing: Remove
TRACE_EVENT_FL_USE_CALL_FILTER logic") changed the bit index so
it makes checking TRACE_EVENT_FL_TRACEPOINT flag failed.  It should
0x20 for newer kernels..

Anyway, this kind of problem can happen at anytime.  One needs to
update the crashtool if some struct or variable name changed.  Maybe
it'd make sense to move the crashtool into the kernel tree?

Thanks,
Namhyung

Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Namhyung Kim]

Hello,

On Tue, Jun 28, 2016 at 06:46:34PM +0200, Rabin Vincent wrote:
> On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:
> > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> > > I'm running some guest machines for kernel development.  For debugging
> > > purpose, I use lots of trace_printk() since it's faster than normal
> > > printk().  When kernel crash happens the trace buffer is printed on
> > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > don't want to reduce the size of ring buffer as I want to collect the
> > > debug info as much as possible.  And I also want to see trace from all
> > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > >
> > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > importantly, I don't want to spend the memory for the crashkernel.
> 
> Assuming you're using QEMU:
> 
> QEMU has a dump-guest-memory command which can be used to dump the
> guest's entire memory to an ELF which can be loaded by the crash utility
> to extract the trace buffer.  This doesn't require kexec/kdump or any
> other support from the guest kernel.

Thanks for the info.  Not requiring kexec/kdump step is a big win for
me.  Although I mostly use kvmtool (lkvm), I'll give it a try.


> 
> It's apparently even possible to run QEMU with the guest memory in a
> file and load that to crash directly, although this is not something
> I've had a chance to try out myself:
> 
> https://github.com/crash-utility/crash/commit/89ed9d0a7f7da4578294a492c1ad857244ce7352

Interesting, I'll take a look but wouldn't it impact the performance?

And even if the crash tool is good, it'd be great if I can work
without it (if possible).


Thanks,
Namhyung