[Patch 0/3] driver data: blktrace pass-through support for device

[Patch 0/3] driver data: blktrace pass-through support for device

[Martin Peschke]

Low-level device drivers might have additional data which complements
blktrace data. For example, zfcp, a SCSI HBA driver, is capable of
obtaining additional latency information per request from HBAs. Those
latencies allow to further break down the overall D2C request latency
provided by blktrace.

We propose an additional trace for blktrace, called "driver data". It is
a sort of small binary blob, which contains a low-level driver specific
struct. Blktrace would be able to filter this trace (-a option) and
include it within its binary output. A small device driver specific tool
on top of blktrace would then analyze "driver data" traces.

Patch 1/3 makes the blktrace kernel code provide blk_add_driver_data()
for use by device drivers.

Patch 2/3 adds support for driver data traces to blktrace tools.

Patch 3/3 makes zfcp provide additional request latency and queue
utilization data through blktrace.


Patches are against scsi-misc and blktrace git.


Martin

Re: [Patch 0/3] driver data: blktrace pass-through support for device

[Alan D. Brunelle]

Martin Peschke wrote:
> Low-level device drivers might have additional data which complements
> blktrace data. For example, zfcp, a SCSI HBA driver, is capable of
> obtaining additional latency information per request from HBAs. Those
> latencies allow to further break down the overall D2C request latency
> provided by blktrace.
> 
> We propose an additional trace for blktrace, called "driver data". It is
> a sort of small binary blob, which contains a low-level driver specific
> struct. Blktrace would be able to filter this trace (-a option) and
> include it within its binary output. A small device driver specific tool
> on top of blktrace would then analyze "driver data" traces.
> 
> Patch 1/3 makes the blktrace kernel code provide blk_add_driver_data()
> for use by device drivers.
> 
> Patch 2/3 adds support for driver data traces to blktrace tools.
> 
> Patch 3/3 makes zfcp provide additional request latency and queue
> utilization data through blktrace.
> 
> 
> Patches are against scsi-misc and blktrace git.
> 
> 
> Martin

Hi Martin -

Have you thought about just using blk_add_trace_msg instead?

Alan

Re: [Patch 0/3] driver data: blktrace pass-through support for

[Martin Peschke]

On Wed, 2008-07-16 at 10:22 -0400, Alan D. Brunelle wrote:
> Martin Peschke wrote:
> > Low-level device drivers might have additional data which complements
> > blktrace data. For example, zfcp, a SCSI HBA driver, is capable of
> > obtaining additional latency information per request from HBAs. Those
> > latencies allow to further break down the overall D2C request latency
> > provided by blktrace.
> > 
> > We propose an additional trace for blktrace, called "driver data". It is
> > a sort of small binary blob, which contains a low-level driver specific
> > struct. Blktrace would be able to filter this trace (-a option) and
> > include it within its binary output. A small device driver specific tool
> > on top of blktrace would then analyze "driver data" traces.
> > 
> > Patch 1/3 makes the blktrace kernel code provide blk_add_driver_data()
> > for use by device drivers.
> > 
> > Patch 2/3 adds support for driver data traces to blktrace tools.
> > 
> > Patch 3/3 makes zfcp provide additional request latency and queue
> > utilization data through blktrace.
> > 
> > 
> > Patches are against scsi-misc and blktrace git.
> > 
> > 
> > Martin
> 
> Hi Martin -
> 
> Have you thought about just using blk_add_trace_msg instead?

Hi Alan,
good point. Yes, we thought about it. It actually inspired
blk_add_drv_data().

blk_add_trace_msg adds information to the textual output of blkparse.
There is no point in forwarding it to the binary output.

We need the opposite. We want driver data to show up in binary ouput.
But there is no point in blkparse trying to generate some ascii message
from low-level device driver specific data.

Besides this would lead us down the path of parsing textual output for
our monitoring tool. This would burn more CPU than a c-program which
consumes binary data.

Martin

Re: [Patch 0/3] driver data: blktrace pass-through support for device

[Alan D. Brunelle]

Martin Peschke wrote:
> On Wed, 2008-07-16 at 10:22 -0400, Alan D. Brunelle wrote:
>> Martin Peschke wrote:
>>> Low-level device drivers might have additional data which complements
>>> blktrace data. For example, zfcp, a SCSI HBA driver, is capable of
>>> obtaining additional latency information per request from HBAs. Those
>>> latencies allow to further break down the overall D2C request latency
>>> provided by blktrace.
>>>
>>> We propose an additional trace for blktrace, called "driver data". It is
>>> a sort of small binary blob, which contains a low-level driver specific
>>> struct. Blktrace would be able to filter this trace (-a option) and
>>> include it within its binary output. A small device driver specific tool
>>> on top of blktrace would then analyze "driver data" traces.
>>>
>>> Patch 1/3 makes the blktrace kernel code provide blk_add_driver_data()
>>> for use by device drivers.
>>>
>>> Patch 2/3 adds support for driver data traces to blktrace tools.
>>>
>>> Patch 3/3 makes zfcp provide additional request latency and queue
>>> utilization data through blktrace.
>>>
>>>
>>> Patches are against scsi-misc and blktrace git.
>>>
>>>
>>> Martin
>> Hi Martin -
>>
>> Have you thought about just using blk_add_trace_msg instead?
> 
> Hi Alan,
> good point. Yes, we thought about it. It actually inspired
> blk_add_drv_data().
> 
> blk_add_trace_msg adds information to the textual output of blkparse.
> There is no point in forwarding it to the binary output.
> 
> We need the opposite. We want driver data to show up in binary ouput.
> But there is no point in blkparse trying to generate some ascii message
> from low-level device driver specific data.
> 
> Besides this would lead us down the path of parsing textual output for
> our monitoring tool. This would burn more CPU than a c-program which
> consumes binary data.
> 
> Martin
> 
> 

Hm, again, if the blktrace | blkiomon (direct, no blkparse) is how you
do things, I would think that it would /not/ take all that much
processing power to parse character strings. (I've heard that IBM makes
some pretty fast processors... :-) ) In particular, if you make the
character strings more machine readable (no text, just numbers) for
example it's just some atoi's going on.

Alan

Re: [Patch 0/3] driver data: blktrace pass-through support

[Martin Peschke]

On Wed, 2008-07-16 at 10:47 -0400, Alan D. Brunelle wrote:
> Hm, again, if the blktrace | blkiomon (direct, no blkparse) is how you
> do things,

no, it's blktrace | blkparse | blkiomon

Isn't blkparse needed to sort all those per-CPU traces by time?
I want this feature.

> I would think that it would /not/ take all that much
> processing power to parse character strings.

Anyway, it's more expensive.

>  (I've heard that IBM makes some pretty fast processors... :-) )

Sure. But we don't make our money from wasting their CPU power :-)

Re: [Patch 0/3] driver data: blktrace pass-through support for device

[Alan D. Brunelle]

Martin Peschke wrote:
> On Wed, 2008-07-16 at 10:47 -0400, Alan D. Brunelle wrote:
>> Hm, again, if the blktrace | blkiomon (direct, no blkparse) is how you
>> do things,
> 
> no, it's blktrace | blkparse | blkiomon
> 
> Isn't blkparse needed to sort all those per-CPU traces by time?
> I want this feature.

The traces should come out pretty close from blktrace itself - I'm
assuming you're interested in putting out statistics in multiples of
seconds. Perhaps some test cases should be tried, but...

> 
>> I would think that it would /not/ take all that much
>> processing power to parse character strings.
> 
> Anyway, it's more expensive.

... it certainly seems to me that if one could remove blkparse
completely you'd save a /tremendous/ amount of CPU at the slight cost of
perhaps having some off-by-a-few counts. (Meaning: You may have a few
C's that you could not match up with D's (because they come out "later"
from blktrace).)

Alan

Re: [Patch 0/3] driver data: blktrace pass-through

[Martin Peschke]

On Wed, 2008-07-16 at 11:15 -0400, Alan D. Brunelle wrote:
> Martin Peschke wrote:
> > On Wed, 2008-07-16 at 10:47 -0400, Alan D. Brunelle wrote:
> >> Hm, again, if the blktrace | blkiomon (direct, no blkparse) is how you
> >> do things,
> > 
> > no, it's blktrace | blkparse | blkiomon
> > 
> > Isn't blkparse needed to sort all those per-CPU traces by time?
> > I want this feature.
> 
> The traces should come out pretty close from blktrace itself - I'm
> assuming you're interested in putting out statistics in multiples of
> seconds. Perhaps some test cases should be tried, but...

I will give it a try

> ... it certainly seems to me that if one could remove blkparse
> completely you'd save a /tremendous/ amount of CPU

sounds tempting to me

> at the slight cost of
> perhaps having some off-by-a-few counts. (Meaning: You may have a few
> C's that you could not match up with D's (because they come out "later"
> from blktrace).)

I had to make provisions for this case in blkiomon anyway.

I suspect blkparse's batch handling to deliver some C's earlier than the
matching D's. But I have never tracked it down.

Thanks,
Martin

Re: [Patch 0/3] driver data: blktrace pass-through support for

[Martin Peschke]

On Wed, 2008-07-16 at 11:15 -0400, Alan D. Brunelle wrote:
> Martin Peschke wrote:
> > On Wed, 2008-07-16 at 10:47 -0400, Alan D. Brunelle wrote:
> > >
> > > Hm, again, if the blktrace | blkiomon (direct, no blkparse) is how
> > > you do things,

blktrace | blkiomon   works out of the box :-)

Re: [Patch 0/3] driver data: blktrace pass-through support for device

[Alan D. Brunelle]

Martin Peschke wrote:
> On Wed, 2008-07-16 at 11:15 -0400, Alan D. Brunelle wrote:
>> Martin Peschke wrote:
>>> On Wed, 2008-07-16 at 10:47 -0400, Alan D. Brunelle wrote:
>>>> Hm, again, if the blktrace | blkiomon (direct, no blkparse) is how
>>>> you do things,
> 
> blktrace | blkiomon   works out of the box :-)
> 
> 

Great - that should remove a lot of CPU processing...

Alan

[Patch 0/3] driver data: blktrace pass-through support for device

[Martin Peschke]

Hi Jens,
may I ask what you think about this approach? Would you consider
including the following patch series in blktrace?

Here is a discussion dating back to the first time I have posted these
patches:
http://marc.info/?l=linux-btrace&m\x121620751101386&w=2


Thanks,
Martin




Low-level device drivers might have additional data which complements
blktrace data. For example, zfcp, a SCSI HBA driver, is capable of
obtaining additional latency information per request from HBAs. Those
latencies allow to further break down the overall D2C request latency
provided by blktrace.

We propose an additional trace for blktrace, called "driver data". It is
a sort of small binary blob, which contains a low-level driver specific
struct. Blktrace would be able to filter this trace (-a option) and
include it within its binary output. A small device driver specific tool
on top of blktrace would then analyze "driver data" traces.

Patch 1/3 makes the blktrace kernel code provide blk_add_driver_data()
for use by device drivers.

Patch 2/3 adds support for driver data traces to blktrace tools.

Patch 3/3 makes zfcp provide additional request latency and queue
utilization data through blktrace.


Patches are against scsi-misc and blktrace git.

Re: [Patch 0/3] driver data: blktrace pass-through support for

[Martin Peschke]

Hi Jens,
this one might have slipped through your fingers.

This small extension would be really useful for us to complement
blktrace data.

What do you think - yes or no, or maybe? 

Thanks,
Martin


On Sun, 2008-10-05 at 15:07 +0200, Martin Peschke wrote:
> Hi Jens,
> may I ask what you think about this approach? Would you consider
> including the following patch series in blktrace?
> 
> Here is a discussion dating back to the first time I have posted these
> patches:
> http://marc.info/?l=linux-btrace&m\x121620751101386&w=2
> 
> 
> Thanks,
> Martin
> 
> 
> 
> 
> Low-level device drivers might have additional data which complements
> blktrace data. For example, zfcp, a SCSI HBA driver, is capable of
> obtaining additional latency information per request from HBAs. Those
> latencies allow to further break down the overall D2C request latency
> provided by blktrace.
> 
> We propose an additional trace for blktrace, called "driver data". It is
> a sort of small binary blob, which contains a low-level driver specific
> struct. Blktrace would be able to filter this trace (-a option) and
> include it within its binary output. A small device driver specific tool
> on top of blktrace would then analyze "driver data" traces.
> 
> Patch 1/3 makes the blktrace kernel code provide blk_add_driver_data()
> for use by device drivers.
> 
> Patch 2/3 adds support for driver data traces to blktrace tools.
> 
> Patch 3/3 makes zfcp provide additional request latency and queue
> utilization data through blktrace.
> 
> 
> Patches are against scsi-misc and blktrace git.
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrace" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html