[Ocfs2-devel] [PATCH 0/2] ocfs2: add trace event support for ocfs2

[Ocfs2-devel] [PATCH 0/2] ocfs2: add trace event support for ocfs2

[Wengang Wang]

following patches add trace event support for ocfs2.
for tracepoint, see Documentation/trace/tracepoints.txt
for trace events, see Documentation/trace/events.txt

the first patch add file include/trace/events/ocfs2.h to kernel tree. it is used
to store trace declarations. it's contents are self-explained.
NOTE that each ocfs2 header file should be above "#include <linux/tracepoint.h>"

the second one is an example which adds trace event to ocfs2_file_aio_read().
NOTE that CREATE_TRACE_POINTS must be defined before including trace/event/ocfs2.h
when trace event is used.

enable/disable the trace event:
1) debugfs is need to be mount for using trace event.
	mount -t debugfs debugfs /sys/kernel/debug(or somewhere else)
2) enable/disable some specific event,
	echo '1'/'0'>/sys/kernel/debug/tracing/events/ocfs2/ocfs2_file_aio_read/enable
3) to see the trace log,
	cat /sys/kernel/debug/tracing/trace

an output example is:
[root at cool ocfs2_file_aio_read]# cat /sys/kernel/debug/tracing/trace
# tracer: nop                                      
#                                                  
#           TASK-PID    CPU#    TIMESTAMP  FUNCTION
#              | |       |          |         |    
             cat-2587  [001]   708.929889: ocfs2_file_aio_read: 0xf544bed8 0xf544bf5c 1 0

             cat-2587  [000]   708.944128: ocfs2_file_aio_read: 0xf544bed8 0xf544bf5c 1 43

examples of using filter.
1) if pid is 0, do the trace:
	echo "common_pid == 0" >/sys/kernel/debug/tracing/events/ocfs2/ocfs2_file_aio_read/filter
2) if group id not 0, do the trace:
	echo "common_tgid != 0" >filter

for filter expression, so far, see http://article.gmane.org/gmane.linux.kernel/887690?

[Ocfs2-devel] [PATCH 0/2] ocfs2: add trace event support for ocfs2

[Wengang Wang]

Wengang Wang wrote:
> following patches add trace event support for ocfs2.
> for tracepoint, see Documentation/trace/tracepoints.txt
> for trace events, see Documentation/trace/events.txt
> 
> the first patch add file include/trace/events/ocfs2.h to kernel tree. it is used
> to store trace declarations. it's contents are self-explained.
> NOTE that each ocfs2 header file should be above "#include <linux/tracepoint.h>"
> 
> the second one is an example which adds trace event to ocfs2_file_aio_read().
> NOTE that CREATE_TRACE_POINTS must be defined before including trace/event/ocfs2.h
> when trace event is used.

and CREATE_TRACE_POINTS should be defined in only one .c file please.

regards,
wengang.

> 
> enable/disable the trace event:
> 1) debugfs is need to be mount for using trace event.
> 	mount -t debugfs debugfs /sys/kernel/debug(or somewhere else)
> 2) enable/disable some specific event,
> 	echo '1'/'0'>/sys/kernel/debug/tracing/events/ocfs2/ocfs2_file_aio_read/enable
> 3) to see the trace log,
> 	cat /sys/kernel/debug/tracing/trace
> 
> an output example is:
> [root at cool ocfs2_file_aio_read]# cat /sys/kernel/debug/tracing/trace
> # tracer: nop                                      
> #                                                  
> #           TASK-PID    CPU#    TIMESTAMP  FUNCTION
> #              | |       |          |         |    
>              cat-2587  [001]   708.929889: ocfs2_file_aio_read: 0xf544bed8 0xf544bf5c 1 0
> 
>              cat-2587  [000]   708.944128: ocfs2_file_aio_read: 0xf544bed8 0xf544bf5c 1 43
> 
> examples of using filter.
> 1) if pid is 0, do the trace:
> 	echo "common_pid == 0" >/sys/kernel/debug/tracing/events/ocfs2/ocfs2_file_aio_read/filter
> 2) if group id not 0, do the trace:
> 	echo "common_tgid != 0" >filter
> 
> for filter expression, so far, see http://article.gmane.org/gmane.linux.kernel/887690?
> 
> 

[Ocfs2-devel] [PATCH 0/2] ocfs2: add trace event support for ocfs2

[Joel Becker]

On Thu, Sep 17, 2009 at 06:19:44AM +0800, Wengang Wang wrote:
> following patches add trace event support for ocfs2.

Wengang,
	This looks like a nice start.  Thank you for getting this going.
I'm sorry I haven't gotten back until now, I've been busy.
	What I'd like to do next is get a small starting set of
tracepoints.  You have one tracepoint in this patch as a good example,
but we should get a few more.  I don't want every mlog changed into a
tracepoint.  Some of them are old and useless :-)  Instead, I'd like to
find information we often want while debugging.
	Tao, Tiger, and Sunil, what are parts of the code you often come
across when debugging?  Certain mlog() prints you always find really
important to see.  If you point them out, Wengang can cook up
tracepoints for them.

Joel

-- 

"But all my words come back to me
 In shades of mediocrity.
 Like emptiness in harmony
 I need someone to comfort me."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127

[Ocfs2-devel] [PATCH 0/2] ocfs2: add trace event support for ocfs2

[Wengang Wang]

Hi all,

Joel Becker wrote:
> On Thu, Sep 17, 2009 at 06:19:44AM +0800, Wengang Wang wrote:
>> following patches add trace event support for ocfs2.
> 
> Wengang,
> 	This looks like a nice start.  Thank you for getting this going.
> I'm sorry I haven't gotten back until now, I've been busy.

Joel, no problem!

> 	What I'd like to do next is get a small starting set of
> tracepoints.  You have one tracepoint in this patch as a good example,
> but we should get a few more.  I don't want every mlog changed into a
> tracepoint.  Some of them are old and useless :-)  Instead, I'd like to
> find information we often want while debugging.

yes, I understand that.

> 	Tao, Tiger, and Sunil, what are parts of the code you often come
> across when debugging?  Certain mlog() prints you always find really
> important to see.  If you point them out, Wengang can cook up
> tracepoints for them.

and yes, welcome to give me suggests for the trace points!!

regards,
wengang.

[Ocfs2-devel] [PATCH 0/2] ocfs2: add trace event support for ocfs2

[Tao Ma]

Hi Joel/Wengang,

Joel Becker wrote:
> On Thu, Sep 17, 2009 at 06:19:44AM +0800, Wengang Wang wrote:
>> following patches add trace event support for ocfs2.
> 
> Wengang,
> 	This looks like a nice start.  Thank you for getting this going.
> I'm sorry I haven't gotten back until now, I've been busy.
> 	What I'd like to do next is get a small starting set of
> tracepoints.  You have one tracepoint in this patch as a good example,
> but we should get a few more.  I don't want every mlog changed into a
> tracepoint.  Some of them are old and useless :-)  Instead, I'd like to
> find information we often want while debugging.
> 	Tao, Tiger, and Sunil, what are parts of the code you often come
> across when debugging?  Certain mlog() prints you always find really
> important to see.  If you point them out, Wengang can cook up
> tracepoints for them.
aops.c maybe? And recovery/dlm(The most awesome part)?

btw, I think the whole mechanism of b-tree(alloc.c) is quite stable now 
after tristan's stress test case for refcount tree. :) So no urgent need 
for this file by now.

Regards,
Tao
> 
> Joel
> 

[Ocfs2-devel] [PATCH 0/2] ocfs2: add trace event support for ocfs2

[Joel Becker]

On Thu, Oct 15, 2009 at 11:05:45AM +0800, Tao Ma wrote:
> Hi Joel/Wengang,
> 
> Joel Becker wrote:
> > On Thu, Sep 17, 2009 at 06:19:44AM +0800, Wengang Wang wrote:
> >> following patches add trace event support for ocfs2.
> > 
> > Wengang,
> > 	This looks like a nice start.  Thank you for getting this going.
> > I'm sorry I haven't gotten back until now, I've been busy.
> > 	What I'd like to do next is get a small starting set of
> > tracepoints.  You have one tracepoint in this patch as a good example,
> > but we should get a few more.  I don't want every mlog changed into a
> > tracepoint.  Some of them are old and useless :-)  Instead, I'd like to
> > find information we often want while debugging.
> > 	Tao, Tiger, and Sunil, what are parts of the code you often come
> > across when debugging?  Certain mlog() prints you always find really
> > important to see.  If you point them out, Wengang can cook up
> > tracepoints for them.
> aops.c maybe? And recovery/dlm(The most awesome part)?

	But what things?  We're not just going to replace every mlog,
and we're not going to do every function.  We really want to figure out
what information is actually useful and add tracepoints for it.  So if
you have particular functions you need data on, or particular mlog()
calls you always like to see, we'd love to hear about it.
	Wengang, dlmglue.c is definitely a place to look at.  I'd love
to see events for the inc/dec of l_ro_holders and l_ex_holders.  I'd
also love to see the mlogs for 'BAST fired' and 'UNLOCK AST fired'
converted, and a similar event for the regular AST.

Joel

-- 

"To fall in love is to create a religion that has a fallible god."
        -Jorge Luis Borges

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127

[Ocfs2-devel] [PATCH 0/2] ocfs2: add trace event support for ocfs2

[TaoMa]

Joel Becker wrote:
> On Thu, Oct 15, 2009 at 11:05:45AM +0800, Tao Ma wrote:
>   
>> Hi Joel/Wengang,
>>
>> Joel Becker wrote:
>>     
>>> On Thu, Sep 17, 2009 at 06:19:44AM +0800, Wengang Wang wrote:
>>>       
>>>> following patches add trace event support for ocfs2.
>>>>         
>>> Wengang,
>>> 	This looks like a nice start.  Thank you for getting this going.
>>> I'm sorry I haven't gotten back until now, I've been busy.
>>> 	What I'd like to do next is get a small starting set of
>>> tracepoints.  You have one tracepoint in this patch as a good example,
>>> but we should get a few more.  I don't want every mlog changed into a
>>> tracepoint.  Some of them are old and useless :-)  Instead, I'd like to
>>> find information we often want while debugging.
>>> 	Tao, Tiger, and Sunil, what are parts of the code you often come
>>> across when debugging?  Certain mlog() prints you always find really
>>> important to see.  If you point them out, Wengang can cook up
>>> tracepoints for them.
>>>       
>> aops.c maybe? And recovery/dlm(The most awesome part)?
>>     
>
> 	But what things?  We're not just going to replace every mlog,
> and we're not going to do every function.  We really want to figure out
> what information is actually useful and add tracepoints for it.  So if
> you have particular functions you need data on, or particular mlog()
> calls you always like to see, we'd love to hear about it.
>   
No idea of recovery/dlm since I don't have much experience in it.
As for aops.c, I think the most important part is ocfs2_write_begin_no_lock.
It is very complicated, sometimes it will write inline_data, sometimes 
CoW a refcount record, sometimes makes a extent written. It is helpful 
for us to know some detail information about it.

Regards,
Tao
> 	Wengang, dlmglue.c is definitely a place to look at.  I'd love
> to see events for the inc/dec of l_ro_holders and l_ex_holders.  I'd
> also love to see the mlogs for 'BAST fired' and 'UNLOCK AST fired'
> converted, and a similar event for the regular AST.
>
> Joel
>
>