Re: [ANNOUNCE][RFC][PATCH] Crash-utility, tracing: enable crash to analyze tracing from core-file (make tracing can act as a flight recorder)

[Lai Jiangshan <laijs@cn.fujitsu.com>]

Peter Zijlstra wrote:
> On Mon, 2009-08-03 at 22:48 +0800, Lai Jiangshan wrote:
>> + /* SIGH, we cann't get "print fmt" from core-file */
> 
> doesn't crash have a vmlinux around to look at?


I don't known how to look at it:

static int								\
ftrace_format_##call(struct trace_seq *s)				\
{									\
	struct ftrace_raw_##call field __attribute__((unused));		\
	int ret = 0;							\
									\
	tstruct;							\
									\
	trace_seq_printf(s, "\nprint fmt: " print);			\
									\
	return ret;							\
}

We need the address of the const string <"\nprint fmt: " print>,
I don't know how to get it current from vmlinux.

I have sent a patch which try to reduce the size of .txt in kernel
and reduce coupling. This patch has a little side-effect:
the "print fmt"'s address is saved in struct ftrace_event_call.

[
http://lkml.org/lkml/2009/7/30/30

This patch is really useful for kernel. We don't change the kernel
even a single line for the purpose of we can read the core-file easier.

This patch is also helpful for crash more or less. So I mention it.
]

> 
>> +/*
>> + * TODO: implement event_generic_print_fmt_print() when the print fmt
>> + * in tracing/events/$SYSTEM/$TRACE/format becomes a will-defined
>> + * language.
>> + */
> 
> (well-defined)
> 
> That would require the running kernel to be the same as the one that
> generated the core file -- not very usable imho.
> 

This "tracing/events/$SYSTEM/$TRACE/format" is core-file's(or the dead
kernel's) "tracing/events/$SYSTEM/$TRACE/format", not the running kernel's.



"print fmt" is generated by the macro TP_printk(), it is not a
will-defined language. Any C-Language code and other weird statement
can be embedded in it. I don't want to induce a C-Language in crash.

A example:
print fmt: "call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d", REC->call_site, REC->ptr, REC->bytes_req, REC->bytes_alloc, (REC->gfp_flags) ? __print_flags(REC->gfp_flags, "|", {(unsigned long)(((gfp_t)0x10u) | ((gfp_t)0x40u) | ((gfp_t)0x80u) | ((gfp_t)0x20000u) | ((gfp_t)0x02u) | ((gfp_t)0x08u)), "GFP_HIGHUSER_MOVABLE"}, {(unsigned long)(((gfp_t)0x10u) | ((gfp_t)0x40u) | ((gfp_t)0x80u) | ((gfp_t)0x20000u) | ((gfp_t)0x02u)), "GFP_HIGHUSER"}, {(unsigned long)(((gfp_t)0x10u) | ((gfp_t)0x40u) | ((gfp_t)0x80u) | ((gfp_t)0x20000u)), "GFP_USER"}, {(unsigned long)(((gfp_t)0x10u) | ((gfp_t)0x40u) | ((gfp_t)0x80u) | ((gfp_t)0x80000u)), "GFP_TEMPORARY"}, {(unsigned long)(((gfp_t)0x10u) | ((gfp_t)0x40u) | ((gfp_t)0x80u)), "GFP_KERNEL"}, {(unsigned long)(((gfp_t)0x10u) | ((gfp_t)0x40u)), "GFP_NOFS"}, {(unsigned long)(((gfp_t)0x20u)), "GFP_ATOMIC"}, {(unsigned long)(((gfp_t)0x10u)), "GFP_NOIO"}, {(unsigned long)((gfp_t)0x20u), "GFP_HIGH"}, {(unsigned long)((gfp_t)0
x10u), "GFP_WAIT"}, {(unsigned long)((gfp_t)0x40u), "GFP_IO"}, {(unsigned long)((gfp_t)0x100u), "GFP_COLD"}, {(unsigned long)((gfp_t)0x200u), "GFP_NOWARN"}, {(unsigned long)((gfp_t)0x400u), "GFP_REPEAT"}, {(unsigned long)((gfp_t)0x800u), "GFP_NOFAIL"}, {(unsigned long)((gfp_t)0x1000u), "GFP_NORETRY"}, {(unsigned long)((gfp_t)0x4000u), "GFP_COMP"}, {(unsigned long)((gfp_t)0x8000u), "GFP_ZERO"}, {(unsigned long)((gfp_t)0x10000u), "GFP_NOMEMALLOC"}, {(unsigned long)((gfp_t)0x20000u), "GFP_HARDWALL"}, {(unsigned long)((gfp_t)0x40000u), "GFP_THISNODE"}, {(unsigned long)((gfp_t)0x80000u), "GFP_RECLAIMABLE"}, {(unsigned long)((gfp_t)0x08u), "GFP_MOVABLE"} ) : "GFP_NOWAIT", REC->node

Yes, I can write a parser to parse them correctly.
But it'll not work probably when new tracepoint is added to kernel.
It's fragile.

Not only crash tool, other tools also want to parse events in userspace.
The fact that TP_printk() is not well-defined makes tracing
is hard to be used at userspace side.

In my code, I use several defined printer for every TRACE_EVENT().
It's stupid, but it works well and it's simpler than a complicated parser.

Thank you for the reviews and comments.

Regards,

Lai