From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752540AbcGAE2E (ORCPT ); Fri, 1 Jul 2016 00:28:04 -0400 Received: from LGEAMRELO13.lge.com ([156.147.23.53]:38226 "EHLO lgeamrelo13.lge.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752477AbcGAE2C (ORCPT ); Fri, 1 Jul 2016 00:28:02 -0400 X-Original-SENDERIP: 156.147.1.151 X-Original-MAILFROM: namhyung@kernel.org X-Original-SENDERIP: 165.244.98.203 X-Original-MAILFROM: namhyung@kernel.org X-Original-SENDERIP: 10.177.227.17 X-Original-MAILFROM: namhyung@kernel.org Date: Fri, 1 Jul 2016 13:27:58 +0900 From: Namhyung Kim To: Steven Rostedt CC: , "linux-kernel@vger.kernel.org" , Minchan Kim Subject: Re: [QUESTION] Is there a better way to get ftrace dump on guest? Message-ID: <20160701042758.GB32617@sejong> References: <20160628062514.GA4674@danjae.aot.lge.com> <20160628095727.436925cb@gandalf.local.home> <20160629005231.GA14525@sejong> MIME-Version: 1.0 In-Reply-To: <20160629005231.GA14525@sejong> User-Agent: Mutt/1.6.1 (2016-04-27) X-MIMETrack: Itemize by SMTP Server on LGEKRMHUB02/LGE/LG Group(Release 8.5.3FP6|November 21, 2013) at 2016/07/01 13:27:58, Serialize by Router on LGEKRMHUB02/LGE/LG Group(Release 8.5.3FP6|November 21, 2013) at 2016/07/01 13:27:58, Serialize complete at 2016/07/01 13:27:58 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jun 29, 2016 at 09:52:31AM +0900, Namhyung Kim wrote: > Hi Steve, > > On Tue, Jun 28, 2016 at 09:57:27AM -0400, Steven Rostedt wrote: > > On Tue, 28 Jun 2016 15:33:18 +0900 > > Namhyung Kim wrote: > > > > > Send again to correct addresses, sorry! > > > > > > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim wrote: > > > > Hello, > > > > > > > > I'm running some guest machines for kernel development. For debugging > > > > purpose, I use lots of trace_printk() since it's faster than normal > > > > printk(). When kernel crash happens the trace buffer is printed on > > > > console (I set ftrace_dump_on_oops) but it takes too much time. I > > > > don't want to reduce the size of ring buffer as I want to collect the > > > > debug info as much as possible. And I also want to see trace from all > > > > cpu so 'ftrace_dump_on_oop = 2' is not an option. > > > > > > > > I know the kexec/kdump (and the crash tool) can dump and analyze the > > > > trace buffer later. But it's cumbersome to do it everytime and more > > > > importantly, I don't want to spend the memory for the crashkernel. > > > > > > > > So what is the best way to handle this? I'd like to know how others > > > > setup the debugging environment.. > > > > Heh, I'd say something helpful but you basically already shot down all > > of my advice, because what I do is... > > > > 1) Reduce the size of the ring buffer > > > > 2) Dump out just one CPU > > > > 3) use kexec/kdump and make a crash kernel to extract trace.dat from > > > > > > That's my debugging environment, but it looks like you want something > > else. > > Thanks for sharing. Yeah, I'd like to know other ways to overcome > this if possible. Since I don't have enough knowledge about this > area, I hope others would have better idea. :) Now I'm thinking about extending the pstore subsystem. AFAICS it's the best fit for my use case. While it only supports function tracer with a dedicated ftrace_ops now, it can be used for ftrace dump IMHO. Does it make sense to add a virtio pstore driver and saves the dump to files on host? Thanks, Namhyung