[patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Jeff Moyer]

Right now, any of the above three drivers will report Q events in
blktrace but no corresponding C events.  Fix it.

Signed-off-by: Jeff Moyer <jmoyer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

diff --git a/drivers/nvdimm/blk.c b/drivers/nvdimm/blk.c
index 9faaa96..90676f3 100644
--- a/drivers/nvdimm/blk.c
+++ b/drivers/nvdimm/blk.c
@@ -19,6 +19,7 @@
 #include <linux/moduleparam.h>
 #include <linux/nd.h>
 #include <linux/sizes.h>
+#include <trace/events/block.h>
 #include "nd.h"
 
 static u32 nsblk_meta_size(struct nd_namespace_blk *nsblk)
@@ -213,6 +214,7 @@ static blk_qc_t nd_blk_make_request(struct request_queue *q, struct bio *bio)
 		nd_iostat_end(bio, start);
 
  out:
+	trace_block_bio_complete(q, bio, bio->bi_error);
 	bio_endio(bio);
 	return BLK_QC_T_NONE;
 }
diff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c
index 368795a..a11c00d 100644
--- a/drivers/nvdimm/btt.c
+++ b/drivers/nvdimm/btt.c
@@ -23,6 +23,7 @@
 #include <linux/ndctl.h>
 #include <linux/fs.h>
 #include <linux/nd.h>
+#include <trace/events/block.h>
 #include "btt.h"
 #include "nd.h"
 
@@ -1195,6 +1196,7 @@ static blk_qc_t btt_make_request(struct request_queue *q, struct bio *bio)
 		nd_iostat_end(bio, start);
 
 out:
+	trace_block_bio_complete(q, bio, bio->bi_error);
 	bio_endio(bio);
 	return BLK_QC_T_NONE;
 }
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 42b3a82..4ea3b9f 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -29,6 +29,7 @@
 #include <linux/slab.h>
 #include <linux/pmem.h>
 #include <linux/nd.h>
+#include <trace/events/block.h>
 #include "pmem.h"
 #include "pfn.h"
 #include "nd.h"
@@ -165,6 +166,7 @@ static blk_qc_t pmem_make_request(struct request_queue *q, struct bio *bio)
 	if (bio->bi_opf & REQ_FUA)
 		nvdimm_flush(nd_region);
 
+	trace_block_bio_complete(q, bio, bio->bi_error);
 	bio_endio(bio);
 	return BLK_QC_T_NONE;
 }

Re: [patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Christoph Hellwig]

On Wed, Nov 09, 2016 at 02:08:33PM -0500, Jeff Moyer wrote:
> Right now, any of the above three drivers will report Q events in
> blktrace but no corresponding C events.  Fix it.

It seems like that trace point should simply go into bio_endio
and be removed from any existing callsite.

Re: [patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Jeff Moyer]

Christoph Hellwig <hch@infradead.org> writes:

> On Wed, Nov 09, 2016 at 02:08:33PM -0500, Jeff Moyer wrote:
>> Right now, any of the above three drivers will report Q events in
>> blktrace but no corresponding C events.  Fix it.
>
> It seems like that trace point should simply go into bio_endio
> and be removed from any existing callsite.

bio_endio is still called for request_fn drivers, so you'd see two
completion events for those drivers if we did that, no?

-Jeff

Re: [patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Christoph Hellwig]

On Wed, Nov 09, 2016 at 02:31:30PM -0500, Jeff Moyer wrote:
> bio_endio is still called for request_fn drivers, so you'd see two
> completion events for those drivers if we did that, no?

We'd see the bio_endio trace in addition to the request one, but
they are at different granularities.  Similar to how on the issue side
we have trace_block_bio_queue for each bio, and trace_block_rq_issue
for each request.

Re: [patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Jeff Moyer]

Christoph Hellwig <hch@infradead.org> writes:

> On Wed, Nov 09, 2016 at 02:31:30PM -0500, Jeff Moyer wrote:
>> bio_endio is still called for request_fn drivers, so you'd see two
>> completion events for those drivers if we did that, no?
>
> We'd see the bio_endio trace in addition to the request one, but
> they are at different granularities.  Similar to how on the issue side
> we have trace_block_bio_queue for each bio, and trace_block_rq_issue
> for each request.

But on the issue side, we have different trace actions: Q vs. I.  On the
completion side, we just have C.  You'd end up getting two C events for
each Q, and that may confuse existing utilities (such as blkparse, btt,
iowatcher, fio, etc), not to mention any scripts built around the
tracepoints, and any users looking at the raw blkparse output.

So, are you suggesting we add another action on the endio side?  If so,
that's a different patch set.  ;-)  If you're suggesting this multiple C
event thing, I'm not on board with that.

Cheers,
Jeff

Re: [patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Christoph Hellwig]

On Wed, Nov 09, 2016 at 02:43:58PM -0500, Jeff Moyer wrote:
> But on the issue side, we have different trace actions: Q vs. I.  On the
> completion side, we just have C.  You'd end up getting two C events for
> each Q, and that may confuse existing utilities (such as blkparse, btt,
> iowatcher, fio, etc), not to mention any scripts built around the
> tracepoints, and any users looking at the raw blkparse output.
> 
> So, are you suggesting we add another action on the endio side?  If so,
> that's a different patch set.  ;-)  If you're suggesting this multiple C
> event thing, I'm not on board with that.

Ok, good point.  It's a little bit annoying how asymetic the tracepoints
are, but fixing it now might cause more harm than it helps.

That being said, it might still be a good idea to have bio_endio call
the tracepoint, we'll just need a __bio_endio to bypass the tracepoints
for calls from the request layer.  That way all bio-based drivers will
automatically do the right thing.

Re: [patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Jeff Moyer]

Christoph Hellwig <hch@infradead.org> writes:

> On Wed, Nov 09, 2016 at 02:43:58PM -0500, Jeff Moyer wrote:
>> But on the issue side, we have different trace actions: Q vs. I.  On the
>> completion side, we just have C.  You'd end up getting two C events for
>> each Q, and that may confuse existing utilities (such as blkparse, btt,
>> iowatcher, fio, etc), not to mention any scripts built around the
>> tracepoints, and any users looking at the raw blkparse output.
>> 
>> So, are you suggesting we add another action on the endio side?  If so,
>> that's a different patch set.  ;-)  If you're suggesting this multiple C
>> event thing, I'm not on board with that.
>
> Ok, good point.  It's a little bit annoying how asymetic the tracepoints
> are, but fixing it now might cause more harm than it helps.
>
> That being said, it might still be a good idea to have bio_endio call
> the tracepoint, we'll just need a __bio_endio to bypass the tracepoints
> for calls from the request layer.  That way all bio-based drivers will
> automatically do the right thing.

OK, I'll look into that.  I'm also still trying to decide whether a
separate endio event would be useful.  Any opinions on that are welcome.
It could show up in blkparse as 'E'.  For btt, I guess we could add a
Q2E column.  I'm not sure C2E would ever be interesting, but maybe?

-Jeff

Re: [patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Ross Zwisler]

On Fri, Nov 11, 2016 at 09:55:10AM -0500, Jeff Moyer wrote:
> Christoph Hellwig <hch@infradead.org> writes:
> 
> > On Wed, Nov 09, 2016 at 02:43:58PM -0500, Jeff Moyer wrote:
> >> But on the issue side, we have different trace actions: Q vs. I.  On the
> >> completion side, we just have C.  You'd end up getting two C events for
> >> each Q, and that may confuse existing utilities (such as blkparse, btt,
> >> iowatcher, fio, etc), not to mention any scripts built around the
> >> tracepoints, and any users looking at the raw blkparse output.
> >> 
> >> So, are you suggesting we add another action on the endio side?  If so,
> >> that's a different patch set.  ;-)  If you're suggesting this multiple C
> >> event thing, I'm not on board with that.
> >
> > Ok, good point.  It's a little bit annoying how asymetic the tracepoints
> > are, but fixing it now might cause more harm than it helps.
> >
> > That being said, it might still be a good idea to have bio_endio call
> > the tracepoint, we'll just need a __bio_endio to bypass the tracepoints
> > for calls from the request layer.  That way all bio-based drivers will
> > automatically do the right thing.
> 
> OK, I'll look into that.  I'm also still trying to decide whether a
> separate endio event would be useful.  Any opinions on that are welcome.
> It could show up in blkparse as 'E'.  For btt, I guess we could add a
> Q2E column.  I'm not sure C2E would ever be interesting, but maybe?

FWIW I think BRD has this same issue where we get block_bio_queue tracepoint
events but not block_bio_complete.  Solving this in bio_endio() would fix that
driver as well.

Where does the Q (bio enqueue), I (req insert), etc. naming show up?  Looking
at a tracepoint trace in perf I don't see that naming.  Is that just a short
hand used between developers, or is it something else?

Re: [patch] nd_blk,nd_pmem,nd_btt: add endio blktrace events

[Jeff Moyer]

Ross Zwisler <ross.zwisler@linux.intel.com> writes:

> FWIW I think BRD has this same issue where we get block_bio_queue tracepoint
> events but not block_bio_complete.  Solving this in bio_endio() would fix that
> driver as well.

Yeah, there are several other drivers that will benefit.

> Where does the Q (bio enqueue), I (req insert), etc. naming show up?  Looking
> at a tracepoint trace in perf I don't see that naming.  Is that just a short
> hand used between developers, or is it something else?

That shorthand appears in blkparse and btt output (tools that are part
of blktrace).

Cheers,
Jeff