tracepoints in ext4 (and/or ext3?)

tracepoints in ext4 (and/or ext3?)

[Eric Sandeen]

As just an initial inquiry, I'm wondering how people would feel about
putting some tracepoints (trace_mark()) into ext[34] for monitoring the
fs behavior.

Good/bad/indifferent?

Thanks,

-Eric

Re: tracepoints in ext4 (and/or ext3?)

[KOSAKI Motohiro]

> As just an initial inquiry, I'm wondering how people would feel about
> putting some tracepoints (trace_mark()) into ext[34] for monitoring the
> fs behavior.
> 
> Good/bad/indifferent?

I think tracepoint on ext[34] is very good idea :)

Re: tracepoints in ext4 (and/or ext3?)

[Theodore Tso]

On Mon, Aug 11, 2008 at 03:18:07PM -0500, Eric Sandeen wrote:
> As just an initial inquiry, I'm wondering how people would feel about
> putting some tracepoints (trace_mark()) into ext[34] for monitoring the
> fs behavior.

I think it's a great idea!  Do you have some specific tracepoints in
mind?

						- Ted

Re: tracepoints in ext4 (and/or ext3?)

[Eric Sandeen]

Theodore Tso wrote:
> On Mon, Aug 11, 2008 at 03:18:07PM -0500, Eric Sandeen wrote:
>> As just an initial inquiry, I'm wondering how people would feel about
>> putting some tracepoints (trace_mark()) into ext[34] for monitoring the
>> fs behavior.
> 
> I think it's a great idea!  Do you have some specific tracepoints in
> mind?

I'll start thinking of some ;)

tracepoints were a hot topic upstream it seemed, so I figured I'd make
sure people were amenable to instrumenting ext$X before I started in.

Thanks,
-Eric

Re: tracepoints in ext4 (and/or ext3?)

[KOSAKI Motohiro]

> On Mon, Aug 11, 2008 at 03:18:07PM -0500, Eric Sandeen wrote:
> > As just an initial inquiry, I'm wondering how people would feel about
> > putting some tracepoints (trace_mark()) into ext[34] for monitoring the
> > fs behavior.
> 
> I think it's a great idea!  Do you have some specific tracepoints in
> mind?

I think two viewpoint exist. An administrator want to
  - performance mesurement
  - which IO aborted it, if any error happend.

In addition, OS vendor want to
  - split out fs problem and device problem

So, candidate of tracepints are
  - jbd activity
  - boundary activity between fs and block layer 


I don't fs expert, it is just idea.

Re: tracepoints in ext4 (and/or ext3?)

[Jose R. Santos]

On Mon, 11 Aug 2008 15:18:07 -0500
Eric Sandeen <sandeen@redhat.com> wrote:

> As just an initial inquiry, I'm wondering how people would feel about
> putting some tracepoints (trace_mark()) into ext[34] for monitoring the
> fs behavior.
> 
> Good/bad/indifferent?
> 
> Thanks,
> 
> -Eric

Good idea, although Im not sure if ext[34] is the best place we should
start putting markers though.

For ext[34], I would start by putting markers on VFS entry points into
ext4 and journal activity.  For these to be useful though, we also need
markers in the following places:

iochedulers
	elv_next_request()
	elv_add_request()
	elv_completed_request()

scsi
	scsi_prep_fn()
	scsi_dispatch_cmd()
	scsi_done()
	scsi_io_completion()

Entry and exit points of all IO system calls.
	AND
The VFS call for these system calls.

bio
	generic_make_request()
	bio_endio()

Scheduler
	idle_balance()  Personally, I find it useful to know when a
			machine goes idle because is stalling on IO


Im sure Im missing something but this should be a good start to be able
to track the life of a pending IO to see where deficiencies lie.

-JRS

Re: tracepoints in ext4 (and/or ext3?)

[Eric Sandeen]

Jose R. Santos wrote:
> On Mon, 11 Aug 2008 15:18:07 -0500
> Eric Sandeen <sandeen@redhat.com> wrote:
> 
>> As just an initial inquiry, I'm wondering how people would feel about
>> putting some tracepoints (trace_mark()) into ext[34] for monitoring the
>> fs behavior.
>>
>> Good/bad/indifferent?
>>
>> Thanks,
>>
>> -Eric
> 
> Good idea, although Im not sure if ext[34] is the best place we should
> start putting markers though.

Sure, but there have already been patches floated for the vfs, and ...

> For ext[34], I would start by putting markers on VFS entry points into
> ext4 and journal activity.  For these to be useful though, we also need
> markers in the following places:
> 
> iochedulers
> 	elv_next_request()
> 	elv_add_request()
> 	elv_completed_request()

I think there's talk of changing the blktrace infrastructure to use markers.

> scsi
> 	scsi_prep_fn()
> 	scsi_dispatch_cmd()
> 	scsi_done()
> 	scsi_io_completion()
> 
> Entry and exit points of all IO system calls.
> 	AND
> The VFS call for these system calls.


http://lists-archives.org/linux-kernel/16489694-lttng-tracepoint-instrumentation-fs.html

> bio
> 	generic_make_request()
> 	bio_endio()
> 
> Scheduler
> 	idle_balance()  Personally, I find it useful to know when a
> 			machine goes idle because is stalling on IO
> 
> 
> Im sure Im missing something but this should be a good start to be able
> to track the life of a pending IO to see where deficiencies lie.

Yep, this all sounds good.

Thanks,
-Eric

Re: tracepoints in ext4 (and/or ext3?)

[Theodore Tso]

On Tue, Aug 12, 2008 at 08:19:54AM -0500, Jose R. Santos wrote:
> 
> Good idea, although Im not sure if ext[34] is the best place we should
> start putting markers though.

I can think of ext3/4 specific markers that would be useful for people
who are tuning our filesystems for performance.  This would include
when we start and end tranactions, when we force a checkpoint, when we
create and, extend, and finish using a handle in the jbd layer.

In the ext4 itself, knowing when we are mapping delayed allocations
would be useful, as well as when we freeze and unfreeze a filesystem
(i.e., for snapshots).

There are a lot of other tracepoints that probably do make more sense
to be put in the VFS layer, although on thing that would be *really*
nice is some semantic sugar in Systemtap or in a Systemtap tapset so
that we only trigger the tracepoints for a particular filesystem.

     	     	     	 	     	   - Ted

Re: tracepoints in ext4 (and/or ext3?)

[Eric Sandeen]

Theodore Tso wrote:
> On Tue, Aug 12, 2008 at 08:19:54AM -0500, Jose R. Santos wrote:
>> Good idea, although Im not sure if ext[34] is the best place we should
>> start putting markers though.
> 
> I can think of ext3/4 specific markers that would be useful for people
> who are tuning our filesystems for performance.  This would include
> when we start and end tranactions, when we force a checkpoint, when we
> create and, extend, and finish using a handle in the jbd layer.

Yep this is the kind of thing I thought of at first ...

> In the ext4 itself, knowing when we are mapping delayed allocations
> would be useful, as well as when we freeze and unfreeze a filesystem
> (i.e., for snapshots).

I could imagine even things like when/where/how big each allocation is,
also maybe some things in the lookup paths...

We might even be able to ditch the mballoc history in favor of
tracepoints if desired?

-Eric

> There are a lot of other tracepoints that probably do make more sense
> to be put in the VFS layer, although on thing that would be *really*
> nice is some semantic sugar in Systemtap or in a Systemtap tapset so
> that we only trigger the tracepoints for a particular filesystem.
> 
>      	     	     	 	     	   - Ted